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CONTEXT OF THIS VOLUME 

This is one in a series of volumes produced by the JTPA EVALUATION DESIGN PROJECT. 



PURPOSE AND PHILOSOPHY 

The P" r P°^ °!' t 5 is . P r "j ect has been to develop a set of evaluation tools that are useful to states and local serww 
areas (SDAs) in judging the way their JTPA programs are being managed and the impact they are having Th<? in,, 
has been to base these analytic and managerial tools on sound program concepts and research methods, ami to . 
hem such that the information obtained is of practical and direct use in improving JTPA policies and programs 
miMaSiolSSiuS JTPA 0 informatlon is also expected t0 make a uni ^ ue contribution to national training ?ols 

It is hoped that these volumes will stimulate and support state and local evaluation efforts in JTPA, and promote more 
consistency than in previous programs with respect to the issues studied and the methods used to investigate them An 
IK ™ J£ ,m l ° encourage the generation of complementary information on program implementation and impact 
that is comparable across states and SDAs. Comprehensive, comparable information is essential to the development of 
a valid and reliable knowledge base for resolving problems and improving programs. It is also required for adjusting na- 
tional training strategies to changing needs and priorities at the state and local level. 

PRODUCTS 

Consistent with this purpose and philosophy, the project has produced a set of materials to assist states and SDAs is, 
fXSSS f.' r P rograms / These are to be useful m planning, designing and implementing evaluation activities. As an 
integrated collection, each set is developed to support comprehensi -e evaluations over the JTPA planning cycle. 

The careful tailoring of these materials to state and local users is appropriate. JTPA represents a new employment and 
training policy shaped not only by the experience of managers and the perspectives of employers, but by scientific assessments 
or previous approaches for addressing unemployment, poverty and other barriers to economic security. In this context 

^H?nn^c°rl J P H^ P a r0 l ramS r alS ° eXp , eCted 10 be j ud 8 ed -' ln fact - the Act's assessment requirements are more explicit 
and sophisticated than those of any employment and training legislation to date. It clearly distinguishes between monitor- 
ing activities, whose purpose is to determine compliance (such as with performance standards) and evaluation activities, 
whose purpose is to determine how a program is being managed and implemented, and the kinds of effects it is having 
on recipients and relevant others. Equally significant, new constitutencies are expected to make these more rigorous 
assessments. States and SDAs now have this important responsibility. It is the first time in the history of employment 
and training programs that the Federal government's evaluation role has been significantly reduced. 

This change affords states and local areas opportunities to influence public policy. It also requires them to assume new 
oversight responsibilities. Program evaluation is expected to become an integral part of the management organizations 
administering, planning and delivering public training services. This is as it should be. The more information available 
at these levels, where changes in organizations can most readily be made, the more effective the management of JTPA 
programs. This project was undertaken in that context. 

The evaluation tools produced by the project have been developed with a sensitivity to the differing needs, interests 
a *tT?TpT? S / atCand l0C t al ? SerS - Th , ey nave been packa 8 ed int0 a si "g' e comprehensive and integrated set of volumes 

T 1 ?.io if Val ? amn 3t the % a ? d \° Cal UveL Tne set contains Panning ^d evaluation guides and issue ™ 
The following volumes are available in the set: w 



Volume 



Author 



ERIC 



I: Overview 

II: A General Planning Guide 
III: A Guide for Process Evaluations 
III Supplement: Some Process Issues at the State Level 
IV: A Guide for Gross Impact Evaluations 
V: A Guide for Net Impact Evaluations 
VI: An Implementation Manual for Net Impact Evaluations 
VI!: Issues Related to Net Impact Evaluations 

A. Issues in Evaluating Costs and Benefits 

B. The Debate Over Experimental vs. Quasi-Experimental Approaches 
VIII: MIS Issues in Evaluating JTPA 

NOTE: Although each of the discrete products listed above is the responsibility of a single author, each seeks to incor- 
En2f« ?SU,lS ? f , J? rofess,onal P eer th . e ma "y excellent recommendations of the advisory group and tSe ?deas 

and suggestions of the numerous practitioners interviewed in the process of developing these materials 
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Project Team 
Deborah Feldman 
David Grembowski 
David Grembowski 
Carl Simpson 
Terry Johnson 
Terry Johnson 

Ernst Srromsdorfer 
Ann Blalock 
David Grembowski 



To further qualify these volumes, Volume III is accompanied by a supplement for state users. This is consistent with 
the significant : differences between states ; and SDAs in the kinds of process issues that are most essential toKudv .The 
volume on net impact evaluations is sufficiently technical, because of the statistical methods involved, that a practical 
manual has been written to accompany it. This guide and manual tend to be more appropriate for states, since relatively 

snKhfffv 5 are t r f^ lr . e V W ^l sis ' H ? wever - 1^. 8 " "«ful to larger SDAs and consortia of sma'ltr 

SDAs which may want to jointly study the net impact of their programs. Regional evaluations, for example, can be very 
productive in providing management information relevant to regional labor markets. Although there is a separate issue 
paper on evaluating costs and benefits, this issue is also covered in the gross impact and net impact guides. In this respect, 
he user benefits from tnree related but different approaches to this important element of program evaluations. Also 
the user should be aware that the Appendix of Volume II includes A Report on a National/State Survey of Local JTPA 
Constituencies. This survey was carried out by Bonnie Snedeker, with the assistance of Brian O'Sullivan, to provide addi- 
tional input from practitioners to the development of the planning and process evaluation guides. 
In conclusion, several expectations have directed the development of these volumes: 



THE GUIDES 

The General Planning Guide 



This guide is to assist users in planning, funding and developing an organizational capacity to carry out process, gross 
outcome, and net impact evaluations and to utilize their results. Separate state and local versions aie available. 



The E'/alt-ntion Guides 
These volumes are to have the following characteristics: 



□The guides are to complement one another. 

•They are to provide information on program management and other characteristics of program implementation, which 
can • 

-Describe the way in which administrative, managerial and service delivery policies and practices operate to affect 

outcomes, as a set of interventions separate from the program's services. 
-Pinpoint the source, nature and extent of errors and biases for which adjustments must be made in gross and net 

impact evaluations. B 
—Help explain the resists of gross and net impact evaluations. 

•They are to provide information on aggregate gross outcomes, and outcomes differentiated by type of service and 
type of recipient, which can: ' ™ 

-Describe relationships between certain implementation modes and service strategies, and a broad array of client and 

employer outcomes. 3 
—Help explain the results of net impact evaluations. 

—Suggest the more important ouicomes that should be studied in net impact evaluations. 

—Help sort out those aspects of implementation that may be most critical to study in process evaluations. 

•They are to provide information on net impact (the program's return on investment), which can: 
—Closely estimate the effect of the program's services on clients. 

-Suggest which services and client groups are most important to study in broader but less rigorous gross impact studies 
-Help identify the decision points in program implementation (particularly service delivery) which may be most 
important to study in process evaluations. 

□The guides are to enable the user to carry out comprehensive assessments of JTPA programs. 

•They are to allow the user to acquire several different perspectives on the same program within a particular time period: 
on program implementation, on outcomes for clients and employers and on net impact. 

•They are to permit the user to interrelate these different kinds of information to gain a wider understanding of what 
is happening in a program and why. 5 

□The guides are to describe approaches and methodologies as consistently as possible, to achieve comparability. 
•They are to define variables and relationships as similarly as possible. 

•They are to define research designs, and methods of data collection and analysis using as similar concepts as possible. 

□The guides are io draw from past research on employment and training programs, as well as seek new approaches and 
methods of specific value in evaluating JTPA at iht state and local level. 

•They are to replicate, to the extent possible and feasible, the issues and measures reflected in Federal monitoring and 
evaluation decisions. ° 

•They are to make selective use of the results of relevant CETA studies, national studies of JTPA, and issue papers 
on JTPA evaluation by national public interest organizations in the employment and training area. 
•They are to rely on the professional literature in applied social research. 
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THE ISSUE PAPERS 

Volume VII contains two issue papers which serve as companion pieces to the preceding volumes on net intact evalua 
tion. The first paper on cost-benefit issues is designed to help users identify, measure Z^^SSSSXZ 
!TnS 3nd benefll l in dele ™ ini "g ^e program'Vreturn on ££££. SSfpSjSS 

SKiS-ft u fferenl r , eSearCh Slralegies associaled with the net impact approach. The final volume on 

MIS issues is to assis users in better understanding how JTPA and other empiovment and training mS^^ 
tion systems can efficiently support the evaluation of program implementation and imp™. ^ «i/onna 

THE SET OF VOLUMES 

The set is integrated but affords flexible use. The user can utilize the entire set for comprehensive evaluations nver 
Bfrt «** or Ion f er P 1 ™* P^iod, or the user can apply the informatioS S 

flS^or CV 311011 Pn0nlieS limeframCS 3nd givCn lhC CXlenl 0f resources ' SmSK 



It should be understood that although evaluation products have been developed for JTPA their basic nrincinles and 
S^£g!£ m0rC br ° ad,y ^ SlalCS and ,0CaI are3S 10 CVaIualC ^ 

GENERAL ACKNOWLEDGMENTS 

The JTPA EVALUATION DESIGN PROJECT was developed and carried out based on the partnershio ohilosonhv 

p o l d u uas e ^ pannersh ; ps sh ? u,d be reco * nized for their «u" SiS^fifflSS 

KfsJS^-T^ ^ • project development anrf coordination partnership; the public-private funding partnership- 
the interdisciplinary design partnership; and the advisory partnership. * pannersmp, 



The Development and Coordination Partnership: 
Washington Employment Security Department 

Project Coordinators: Gary Bodeutsch, Ann Blalock 
Assistant Coordinators: Deborah Feldman, Chris Webster 
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Carl Simpson 

Bonnie and David Snedeker 
Ernst Stromsdorfer 
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Area of Expertise: 
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Planning 
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Management 
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Design Partnership 
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A. Issues in Evaluating Costs and Benefits 



By Ernst Stromsdorfer 

March 1986 
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ISSUES IN EVALUATING COSTS AND BENEFITS 



INTRODUCTION 

This volume contributes to the analysis of the JTPA program Impacts by 
describing the rationale and procedures for estimating JTPA program 
costs and by showing how costs and benefits are related 1n a human 
capital Investment framework. This Investment framework then becomes a 
management tool to Increase the efficiency of JTPA operation. The 
justification for this approach 1s contained 1n Sec. 106 of the JTPA 
which states that the Congress finds that-- 

(1) 1t 1s essential that criteria for measuring the return on 
this Investment be developed; and 

(2) the basic return on the Investment 1s to be measured by the 
Increased employment and earnings of participants and the 
reduction In welfare dependency." 

Thus, Congress treats the JTPA as a major effort to enhance the labor 
market productivity of program-eligible Individuals by Increasing their 
human capital. For Title IIA, this 1s to be achieved by a variety of 
means as set forth 1n Sec. 204, but all of them have the Intended 
effect of either directly or Indirectly Increasing the human capital of 
program participants and thereby maintaining, Improving or restoring 
their capacity to function effectively 1n the labor market. 

It must be recognized, however, that the benefits that are Intended to 
flow from the program services or treatments described 1n Sec. 204 are 
achieved only by the well-considered and effective expenditure of 
society's resources. In short, for every benefit, there 1s a cost. 
And, 1n general terms, for the cost outlay of the treatment to have 
value and purpose f or society, benefits must equal or exceed costs . 

Benefit-Cost Analysis versus Effectiveness-Cost Analysis. 

The primary benefits of the JTPA are of two kinds: monetary and 
non-monetary. The monetary benefits, such as Increased before-tax 
earnings or reductions 1n welfare payments due to successful 
Integration Into the labor market, are compared against monetary costs 
1n a benefit-cost framework. Non-monetary benefits, such as Increased 
self-esteem measured by some psychological scale or decreased welfare 
dependency measured either as a percent, a probability or total weeks 
on welfare, for Instance, are compared against monetary costs 1n an 
effectiveness-cost analysis. 

The two approaches are conceptually equivalent and the technical 
approaches required for a valid estimate of either are 1n principle the 



er|c io 



same^ Neither Is, In principle, technically superior to the other- 
nna? e nh S ne " ssar1 * ™ r f Practical or easier to Implement As a 
lb* nr If rvat1 °\ we should stress that data limitations, such a> an 

♦ of c ruc1al cost or benefit dat do no . 

methcd to dominate the other. Each, for Instance, ideally requires 
money cost estimates since there 1s no other common denominator of 
cos.s. Using the benefit-cost or the effectiveness-cost ratio the 
measures of net human capital Investment return take the~foTlow1nq 
general form: 1 3 

1. Benefit-Cost : 

Present Value of Total Honey Benefits . Total Benefit-Cost Ratio 
Present Value of Total Money Costs 

2. Effectiveness-Cost : 

Any Effectiveness Mea sure Expressed as a Total = Total Effective- 
Present Value of Total Money Costs ness-Cost Ratio 

It 1s mostly a matter of policy focus or need that will determine which 
cL.*?f a c ppr , oac t nes t0 use - for Instance, the JTPA speaks 

specifically of "return on Investment.". To estimate a return on 
Investment requires that one estimate a benefit-cost ratio, an Internal 
rate of return on Investment or the net present value of an 
1nvestment-~the sum of discounted benefits minus the sum of discounted 
C k°1: S :, How " ver . JTPA also has as a goal the reduction of the number of 
Individuals who are dependent on cash welfare payments. An 
effectiveness-cost ratio would be employed 1n this case to measure for 
example, the decreased probability of receiving any cash welfare 
payment during the year after end of program treatment per $1,000 of 
expenditure of JTPA funds for, say, classroom training. 

However, we recommend that benefit-cost analysis always be attempted 
Then, one can expand the analysis, time and resources permitting, with 
effectiveness-cost analysis. The overall summary measure of program 
Impact-the Increase 1n total before-tax earn1ngs--g1ves policy 
precedence to the use of the benefit-cost approach 



the: benefit-cost evaluation framework 

As noted above, benefit-cost analysis 1s a technique for evaluating 
socla Investments and judging the relative desirability of competing 
social policies. In Its typical application, the technique would be 
used by a government agency to judge a program's efficiency or 
effectiveness to help decide whether the program should be expanded or 
reduced n size and scope, redirected toward goals 1t night achieve 
more easily or discontinued altogether. As a management technique, 
benefit-cost analysis 1s not so much a fixed course of action as a 
broad evaluative approach to addressing complex questions of resource 

1 Sef Exhibit 7 for a precise mathematical definition of these 
concepts . 
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allocation, Including the understanding of the structure and size of 
benefits and costs. Properly executed, such analysis can aid 
significantly in program management. There are certain jeneral 
procedures that are common to all benefit-cost evaluations. 

The first step 1n an evaluation is to define the set of progran 

oenents and attach a dollar value or some non-monetary value to them 

The benefits will, of course, be program-specific, and frequently will 

nnrnnc^c 2 on the basis of a program's legislatively established 

t and ^° be 5 t1ve5 ' In the case of government-sponsored 
employment and training programs, the measures specified in the JTPA as 

hn? 9 n a thor OUtCOmeS , 3re , USUally s P ec1f1ed as a program's main benefits, 
but other economic and non-economic benefits can also be considered as 
part of a more comprehensive evaluation. As noted further below, a 
major concern here is to avoid double-counting. For instance, hours 
worked are a component of earnings (hourly wage rate x total hours 
worked = total before-tax earnings), so in a program evaluation one 
cannot claim that an Increase in hours worked and an Increase 1n 
earnings can be simultaneously counted as benefits. 

The second step is to Identify and estimate program costs. For an 
employment and training program, the most obvious costs are the direct 
costs of operating and administering the program. Most of these are 
incurred at the SDA level or among Us subcontractors. Some costs 
mainly administrative, will occur at the state level. In addition a 
comprehensive calculation of costs should also include the costs that 
an individual incurs when he or she participates in the program. These 
Include the direct costs of participating in a program treatment such 
as the purchase by the participant of books, tools or special 
clothing. A potentially large component of participant costs are the 
earnings the person may have forgone by entering the program rather 
than remaining in the labor market and working at some job. 

The third step is to recognize that the costs and benefits to 
investment 1n human resources occur in different time periods: it is 
therefore necessary to adjust the dollar values of each into present 
terms using an appropriate discount rate; compare their total values- 
arid determine the social efficiency of the program, based on the 
difference in the total dollar value of these costs and benefits If 
the present value of money benefits exceeds that of costs, then the 
program would be judged worthwhile from the perspective of economic 
efficiency; In the opposite case, the program would be iudged 
inefficient, if all costs and benefits that exist have been accounted 
for and they are all measurable in money terms. 2 

Discounting As noted above, the nature of investments in human 
capital is that their costs and benefits occur over time. Before-tax 
earnings benefits to adult female participants in the original Manpower 
Development and Training Act of 1962 were shown to be large and 
positive for more than five years after the participants left the 

2 Clearly, as already noted, costs and benefits occur in both money 
and non-money components as specified in the JTPA. The above money 
benefit-cost approach, taken alone, therefore is subject to error 
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P P am < Ashenfelter, 1978). Similar post-program benefit patterns 
T , , ? men Wh0 P art1c ^ated in the Comprehensive Employment 
reLr- th" 9 * A f C h* K The d J SDersal of "sts and benefits over time 
requires that they be made commensurable in terms of their present 
value since future benefits are worth less than benefits one has ir, 
the present. Likewise, costs that occur in the future have a smaller 
present value, a simple analogy will show why this is so. 

trhn* n „ SnA h 1 Ve a .T °f >1.000 today. If the SDA spends it all on 
training services today, it spends exactly $1,000. The SDA, however, 
cou d postpone this expenditure and put the funds in a bank. This bank 
will offer to pay interest on the deposit of, say, ten percent per 
year. If the SDA leaves the funds in the bank for a year, in one year 
it will nave $1,100. The same $1,000 now is worth $1,100 a year from 
now. Thus, a dollar now is worth more than a dollar in the future 
because present dollars, via interest payments, grow into a larger 
future sum. Thus, present and future dollars cannot be directly 
compared unless the future dollars are made to reflect the same value 
as the present dollars. This is done by discounting a future stream of 
dollars (whether costs or benefits) to their present value. 

The basic equation for estimating a present value is 

PV (present value) = 2 B t 

where t=0 (It) 11 

B is the benefit (cost) in any time period; say a year; 

r is the interest rate used to discount; 

t is any year; and, 

T is the total time period, e.g. 13 years. 

Note that there are three dimensions to the discounting process: 

1. What are the benefits or costs in each time period; 

2. What is the interest rate to be used in discounting; and, 

3. What is the time period over which benefits and costs occur? 

Benefits and costs can be directly estimated for those time periods for 
which complete data exist. If it is assumed that some benefits or 
costs will occur for time periods beyond that for which there are data, 
then assumptions have to be made about when these benefits will occur 
and how large they will be in any time period. Informed judgment is 
the best guide in this exercise since there is almost no information on 
the time profile of benefits and costs to subsidized employment and 
training programs beyond six years after a participant exits such a 
program. 

Choice of an interest rate to be used in discounting is not as easy as 
it would seem at first blush. There are competing theories as to how 
to select the correct interest rate. It is not necessary to discuss 
these theories here except to note that the rate will differ for 
society as a whole compared to, say, a state government or an SDA. The 

r tA *°u* s ° c1ety as a wnole 1s net °f any effect due to inflation 
while this is not the case for a local government or agency. The most 
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practical guidance the choice of Interest rate 1s to select that 
rate at which the local government or agency can lend \ti funds. The 
accounting office of a local government or agency can provide that 
figure. Currently, lending rates, are 1n the neighborhood of ten 
percent, which 1i a relatively high number. 

The time period over which benefits and costs can occur cannot exceed 
the average remaining life expectancy of the trained participants. 
Historically, the average age of trainees has been In the low 30s 
This Implies a 30 year period or so for which benefits (costs) can 
occur. This Is a very long period. Normally, we would expect the 
value or productivity of training to depreciate over time, just like 
the value of any other Investment. And, of course, the value of job 
placement services could depreciate rapidly, depending on the nature of 
the placement process. 

The best advice to offer In these cases 1t to conduct sensitivity 
analyses to see how sensitive the estimate of net. program benefits are 
to the size and duration of future costs and benefits and the Interest 
rate used In discounting. In particular, the higher the Interest rate 
useu, the less Important the other assumptions become. For Instance, 
at a ten percent Interest rate a dollar received 25 years from now Is 
worth only 9.2 cents. At a four percent Interest rate, however, this 
discounted dollar Is worth 37.5 cents. 

Combining Monetary and Non-monetarv Benefits and Costs , other Issues 
arise In the measurement and valuation of benefits and costs. In 
evaluating a training program, some benefits, such as Increased 
before-tax earnings, result from greater work productivity that can be 
both measured and. assigned a dollar value. However, other benefits, 
such as psychic benefits cannot easily be assigned a dollar value, If 
at all. The problem for the analyst Is how to Include these 
nonquantlf lable and nonmeasurable benefits In the analysis, since, 
without them, the analysis 1s Incomplete and potentially misleading. 

The practice employed to address this problem and to make the analysis 
as comprehensive as possible Is to specify In dollar terms all benefits 
and costs measurable In dollars; then one should qualify the results 
with a discussion of measured non-economic outcomes (such as 
educational performance scores) and nonmeasurable outcomes (such as 
Improved work habits) that would affect the overall benefit-cost 
calculation. After these nonquantl f lable and nonmeasurable effects 
have been Identified, the program's effectiveness would be assessed at 
a more Implicit level using this broader concept of benefits and 
costs. As with the narrower analysis, If the estimated difference 
between discounted benefits and costs under this broader concept 
exceeds zero, the program treatment would be judged to be efficient; If 
less than zero, It would be judged Inefficient or Ineffective. 

The Focus and Perspectives nf Benefit-Cost Analysis . As Indicated 
above, the purpose of a benefit-cost analysis Is to judge the social 
efficiency of a program. It asks the question: "Is the value of the 
goods and services and other outcomes available to society greater as a 
result of the program, or would the value have been greater had the 
resources been put to alternative uses?" As an aggregative measure 
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IDENTIFICATION AND MEASUREMENT OF BENEFITS AND COSTS 

The taxonomy of benefits and costs from the perspective of society the 

program participant and the non-part1c1pant (taxpayer) 1s outlined 1n 

Exhibits 1 and 3. Benefits are discussed first, followed by a 

discussion of costs. Key measurement Issues are discussed when 
appropriate. 



A GENERAL TAXONOMY OF BENEFITS 

Benefits to a government subsidized program of employment and training 
such as the JTPA can occur during and after the program treatment. As 
with costs, there are benefits to the participants and the 
non-part1c1pants, the sum of which equals social benef1ts--a full 
economic accounting of benefits. In addition, within social benefits, a 
subset of benefits accrues to employers. These are discussed 1n turn 
Exhibit 1 sets them forth schematically. 

Increased Output during Program Participation . These before-tax 

benefits can occur for two reasons. First, 1f a participant 1s 1n an 
on-the-job (OJT) training program, the value of the output he or she 
creates during that training 1s a social benefit. In general, the 
value of output 1s measured by the earnings paid to the worker, net of 
any program subsidy due to the OJT program treatment. The value of the 
output 1s defined as the net value added created by the worker. This 
value added due to labor 1s the reason why a worker 1s hired and 1s the 
basis upon whlcn the worker 1s paid. Ideally, the worker receives 1n 
earnings the full share of value added created by him or her. Note 
also that this value added 1s a benefit to the employer for which the 
employer 1s willing to make a payment 1n return for Its creation by the 
worker. As a practical matter 1t 1s very hard to Identify that portion 
of an OJT worker's total wage that 1s a subsidy to pay for training and 
that portion which is compensation for the value added created by the 
worker during the OJT process; a worker will create some valuable 
output 1n the process of being trained. No one to our knowledge, has 
ever successfully unraveled this problem for government subsidized OJT 
projects . 

Second, in a classroom training situation, such as an auto repair 
class, output is created that can be sold or, 1f not sold, its value 
can be Imputed. While this may not be a large quantity, it 1s worth 
mentioning in order to maintain completeness in the accounting of 
benefits . 

Increased Output after Program Participation . This 1s by far the 
largest measurable component to post-program benefits and the one with 
which we are most familiar. However, there is considerable confusion 
as to how it 1s properly measured. In particular, as discussed below, 
there is a tendency to double-count Increases in post-program output. 
The manner 1n which we have broken down the components of post-program 
output is designed to avoid such double-counting. 

First, note that the best summary measure of net post-program output is 
the Increase in before-tax earn1ngs--1tem 2d 1n Exhibit 1 The 
measurement of this quantity is central to the analysis of program 

7 



oerore-tax earnings 1s the best summary measure because, 

1. If the labor force participation rat« Increases, other things 
equal, total earnings will Increase; 

2. If total hours worked Increase (unemployment decreases; time to 
find a new job decreases), other things equal, total earnings 
will Increase; and, 

3 " lLZ he * nourl y w 396 r , ate 1ncreases (or any other related measure 
such as monthly salary Increases), then total earnings will 
Increase. 

In summary, any Increase 1n labor force participation; any reduction In 
unemployment (Increase In hour, worked) from whatever source such as 

Z f£ hnTril 0 Mnd 3 l° b ,° r re l Uced Job trover; and, any Increase 
in the hourly wage rate (or Its equivalent) will Increase total 
before-tax earnings. While It 1s sometimes Important for policy 
purposes to decompose the component causes of any earnings Increase, it 
is not absolutely necessary to do so. and often, due to data problems 
or cost and time constraints, 1t cannot be done. Thus, total 
before-tax earnings 1s the most versatile as well as the key summary 
measure of Increase 1n post-program output. 

Since govsrnments, agencies and taxpayers have an Interest 1n the 
measure, we have Included as a benefit Increased Income and "*cise 
ftrfrlc / * e l 6 1*1* c benefUs t0 non-participants are the result of 
increased total before-tax earnings and Increased expenditures caused 
by the Increase in total after-tax earnings, respectively. Again 
these Increased taxes are benefits to non-part1c1 P ants and costs to 
participants For society as a whole, these two sums net to zero. 
Soc ety 1s only Interested In the total Increase In output. Members of 
society— both participants and non-part1dpants--can then agree to 
divide up that Increase 1n output any way they see fit; that 1s thev 
can agree to tax each other. 

Note that 1t will very seldom be the case that net tax gains will equal 
or exceed program costs. Indeed, the correct measure for the return on 
such an Investment 1s fundamentally the total before-tax Increase In 
output (plus other measures of well-being) relative to total social 
costs- all resources used (cost Incurred) to operate the program. 
There 1s often a temptation to relate social costs to non-part1c1pant 
(taxpayer) benefits. This 1s an error since 1t 1s akin to contrasting 
apples with oranges. As can be seen from Exhibits 1 and 3, the social 

H?Jor n tT u " Part1ClP ^ nt C T °^ tS are def1ned differently. They cannot be 
directly compared. Indeed. non-part1c1pant benefits and costs are a 
Part of social benefits and costs. 

Reduction In Welfare Dependenc y. There can be a reduction In welfare 
payments either because one has a lower propensity to qualify for 
welfare benefit level held constant, or the benefit level could drop 
propensity to use welfare being constant. Changes 1n total welfare 
payments represent benefits and costs to participants and 
non-part1c1pants but they net to zero for society since they are 
transfer payments. The true gain to society comes from the Increase 1n 
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Exhibit 1 



A GENERAL TAXONOMY OF THE BENEFITS 
OF THE JOB TRAINING PARTNERSHIP ACT: 
AN ACCOUNTING PERSPECTIVE 



Social 3 Participant 



Non- 
Partldpant 



1. Increased output during program participation-- 
measured by before-tax earnings paid to the 
participant, which Is the value added by the 
worker. 

2. Increased output after program participation. b 

a. Increased probability of being a member of the 
labor force 

b. Increased hours (weeks, months) worked, given 
one's labor force participation 



c , 
d, 



Increased before-tax wage rate per hour worked + 

Increased before-tax earnings (the product of 
2b and 2c above) * 

e. Increased Income and excise taxes paid as a 
result of Increased before-tax earnings and 
consumption from Increased after-tax earnings o 

Reduction 1n welfare dependency 

a. Reduced probability of being on welfare + 

b. Reduced payments, given one's probability of 
being on welfare 0 

Other benefits 

a. Psychic benefits of being a more productive 
member of society, e.g. Increased self-esteem + 

b. Increased or Improved physical and mental 
health, Including such things as reduced drug 
abuse, reduced suicide, reduced spouse or child 
abuse x 



c, 
d. 



Improved work attitudes and work discipline, 

Reduced crime against property and persons 
outside of the family group 



1. 
11 



property 
persons 



o 



o 

0 



0 
0 
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Non- 
Social 3 Participant Participant 

5. Employer benefits 

a. Increase In value added. b + + + 

b. Employer benefits not Included In Increased 
output after program participation^ 

1. Reduced job search costs; this will be due 
in part to more efficient labor market 
Institutions and information brought about 
by the JTPA and reduced job turnover due to 
better worker/training/job matching 

— A quicker job match, with the average 

quality of the job match unchanged + + o 

— An Improved match, with the average 

time for a job match held constant + + o 

11. Reduced training costs due to reduced 
job turnover. (The less often a person 
switches jobs, the less likely the person 
will have to retrain, either formally In 

class or Informally on-the-job.) + + o 



Notes: * y he sum of participant and non-participant benefits equals social 
benefits. Social benefits represent the total Increase in Economic and 
non-economic benefits. 



Note that the Increased total earnings of the participant represent the 
value of that participant to the employer. This value added Is the valu! 
of the extra output created by the worker and for which the employer 
hires and pays him or her. p J 
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total output. This Increase In total output 1s what enables one to 
depend less on welfare or pay others more welfare. Thus, changes In 
welfare payments are neither social costs nor social benefits. 

However, there 1s an Intrinsic value to society, the participant and 
the non-part1dpant to a reduction 1n the dependence of society's 
members on welfare, es long as no one 1s made worse off as a result 
Thus reducing the degree of welfare dependency, other things equal, 1s 
a social benefit. 



Other Benefits 

Psychic benefits, while not directly measurable, exist 1n fact and must 
be noted. They are reflected 1n such concepts as Increased 
self-esteem. Psychometric scales can be used to provide an Index 
measure of this self-esteem, 1f desired. Note that total earnings will 
Increase 1n part due to Increased self-esteem, so there 1s a 
possibility of double-counting when considering any program benefit 
that has psychological or physical well being dimensions. Yet, 1t 1s 
clear that there are gains to Individuals and to society from the 
Increase 1n self-esteem alone. Increased self-esteem 1s valued 1n Its 
own right. 

Improved Physical and Mental Health . It 1s well known that business 
cycles Induce unemployment and economic distress and that these factors 
1n turn lead to reduced physical and mental health, Including Increased 
suicide. Reversing this process can Improve mental an physical health 
The gains are directly Internalized 1n the person and there 1s a 
reduction 1n social resources required to treat these Illnesses Induced 
by economic adversity. Again, the Increase 1n total earnings will pick 
up some of the Improvement 1n mental and physical health— absenteeism 
will be less, for Instance, and productivity while on the job will be 
higher. But, as with self-esteem, these benefits to the quality of 
life add to well-being Independent of their economic contribution. 

Improved Work Attitud es and Work Discipline . A properly designed and 
executed employment or training program can change behavioral habits as 
well as add to human capital. This positive change 1n habits has 
Intrinsic personal and social value 1n Itself. In addition, of course 
such positive behavioral change will also be reflected in Increased 
total before-tax earnings. Getting to work on time, following orders, 
reducing absentee1sm--are all reflected 1n higher earnings. 

Reduced Crime. Ironically, the reduced gains from crime 1s a cost to 
the Individual participant. There are two aspects to reduced "crime. 
First, crime against property represents an Illegal transfer payment--a 
fancy term for theft. Thus, property crime— 1 ndependent of losses due 
to damage-nets to zero for society. However, the legal system and all 
other avoidance costs represents a drain on society's resources If 
crime drops, the use of these resources drops, which 1s a social 
benef It. 

Crime costs are very hard to directly measure. The best efforts to 
date have been 1n the Hathematlca Policy Research Inc. (MPR) study of 
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Employer Benefits 



I^n< na f tUre ° f empl °y er benefits 1s not well understood. The emnlover 
gains from a program like JTPA 1n several ways. These are: empl °y er 

1. A reduction 1 R job search costs both for the Initial Mtp an n 
because of (Ideally) reduced job turnover due to mVre efficient 
matching of worker, training, job and firm. erncient 

2 " rJ,Zt U «Vl\ 1n J ° b tra1n1n 9 costs, both initially and due to 
reduced job turnover as a result of more efficient matching. 

A word must be said here concerning specific and general tralnlno ThP 
employer gams from reduced job training costs' if the tralnlno Ts 

n n rm "h S p PeClf1 H C -, the Sk111 f0r Wh1ch a Participant is trained literal y 
can be used In no other firm except the firm In question Such 

2£^ , sS%?. r fn V 1 1 n d a 8d f? J 7 A ' kV ^* t0 the'firm, not t e 
training, 1f not subsidized, 1s a cost to the firm but 

Sit J£ WOrker - , In P^ndple, the worker will not contribute to the 

11 efUs fToTl Th a p 1n M 9 beCaUS f the WOrker w111 «•* none of III 
Denerits from It. The firm captures all of the be^Tflts tn 

flnn-speclflc training. Historically, so"^ government subsldlzeS 

employment and training projects have subsidized fZ 'specific training 

&S!E% ^^Vl Wl « P the^fTrm-^nSS? 

"aVpVr^osJSvn- *r s V of tVa ™ n 9 when p^-3"s 

all JTPA tralnlnn « Y ° W ° rk6r ' n0t the f1rm - H ° St ' but not 

ex stence of lin^i V* fT'V" natur *- OJT programs reflect the 
n r n , i tra1 ? n9 When 3 Worker 1s P a1d Jflwer wages during 

Irlmii P rJ , Th6Se l0Wer wages are reflections^- the cost of 
training. The worker pays the cost through lower waqes The firm t..«t 
happens to be the locus of training. When the training ends wage 
rise to those of the typical worker In that occupation 9 

SDecm c ml9 tr^n?! eCt ' " V ° ften hard t0 measure whether there 1s any 
fa g But °1t d C 0 p7 1n9 : f SinC I U Can occur a longs1de general 
bene? "'to the firm ' ^ ^ redUCt1 ° n 1 " the C0St 1s a 



Value Added 1s the third major benefit to the firm- however this i< 
he value added that 1s contributed by the worker to the n r m a d 

SJmTc 6 i r S - ^f? ° f the t0tal 0utput created ^ the firm when 
combines land, labor and capital m production. The wage rate 1s the 
direct measure of this value added. Total earnings Is the total 
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increase in value added by a worker over a given time period. Of 
course, this value added, in the form of total before-tax earnings, is 
also a measure of social benefit. Therefore, while the reduction in 
job search costs and specific training costs to the firm can be added 
directly to total social benefit, value added cannot . To do so would 
be double-counting. Nevertheless, it is important to understand the 
concept of value added. It is the reason why firms hire workers in the 
first place. It is what firms pay wages for and it is the reason why 
such wages can be paid at all. 

Double-Counting. The problem of double-counting needs mentioning 
because it is a common problem area and source of error in benefit-cost 
analysis. This is a particular problem when effectiveness-cost ratios 
are used or when they are combined with a benefit-cost measure. 

As noted in the beginning of this chapter, there is no fundamental 
conceptual difference or methodological advantage between benefit-cost 
analysis and effectiveness-cost analysis. Any given project or program 
treatment can be jointly evaluated by both approaches. However, as 
Exhibit 2 shows, many cf the most important effectiveness measures are 
a component of the fundamental program outcome of total before-tax 
earnings over some period of time , usually a year. 

Consider the following: The JTPA is designed to improve the economic 
condition of the legally-defined target populations. These people are 
generally poor, either due to temporary shocks to their income (e.g., 
displaced homemakers) or earnings capacity or due to more permanent 
problems such as lack of English speaking ability. To relieve this 
relative poverty, the main problem is to increase their income, the 
major component of which is earnings. To increase one's earnings, the 
following sequence of events must occur as a result of a JTPA treatment: 

• One must enter the labor market. That is, there must be labor 
force participation, or nothing else will happen. 

• Having entered the labor market, one must become employed. One has 
to get a job. 

• Having become employed, one must work more hours per week, holding 
weeks worked constant, or one must work more weeks per year, 
holding hours constant, or both hours and weeks worked can increase. 

• One's wage rate must be positive. Ideally, one's wage rate will 
increase over what it would have been had one not entered the JTPA 
program. This is a clear-cut measure of program success, but not 
the only measure. 

• Finally, earnings is the product of hourly wage rate times total 
hours worked. 

Clearly, an Inspection of Exhibit 2 and a review of the other types of 
benefits listed in Borus and Tash (1970) show that many, if not most, 
of the benefits listed are contained in and are a total or partial 
result of improved earnings. Thus, improved housing or health can 
occur if earnings increase so one can afford to purchase more of 
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Exhibit 2 

EXAMPLES OF PARTIAL OR TOTAL DOUBLE -COUNTING 

IN THE MEASURE OF BENEFITS 
TO EMPLOYMENT AND TRAINING PROGRAMS 

Type of Benefit Contained In a Measure of Total Before-Tax 
lype or benefit Earnings 1n Whole or 1n Part? 

• Reduced unemployment Yes 

• Increased employment Yes 

• Increased productivity of the 

labor force--the ability to 

earn higher hourly wage rates Yes 

• Improved equity 1n the distribution of: 

- Tncome 

- Employment 

• Increased labor force participation Yes 

• Increased self-esteem Yes 

• Increased job satisfaction Yes 

« Increased work week for those 

desiring more work Yes 

• Reduced unnecessary turnover Yes 

• Increased mobility of the labor force Ye s 

• Reduced dependency on government Yes 

• Reduced time to find a job Yes 

• Improved physical health Yes 

• Improved housing Yes 

• Improved skills and education Yes 



Adapted from Borus and Tash (1970). Table 3. 



Yes 
Yes 
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Exhibit 3 



A GENERAL TAXONOMY OF THE COSTS OF 
THE JOB TRAINING PARTNERSHIP ACT: 
AN ACCOUNTING PERSPECTIVE 



Non- 
Social 3 Participant Partldpanl 

1. Program operating cost 

a. Overhead and start-up costs 

(fixed costs) _ 0 

b. Variable costs of program operation 0 

2. Administrative cost above the project 
level 

a. Overhead costs q 

b. Variable costs q 

3. Participant opportunity costs 

a. Foregone earnings plus 

fringe payments - 0 

b. Foregone non-market activities - 0 

4. Increased class attendance costs 

a. Transportation - 

b. Child care _ ~ 

c. Other (except psychic costs) - 

5. Psychic costs of participating 

In training programs - 0 

6. Income maintenance transfers (stipends) 0 + 



Note: a The sum of participant and non-participant costs equal social 
costs. Social costs represent full economic costs. 
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J5lf p<t..mV services As noted above, better mental health or 
a result of getting any job or getting a better job. In 

hIZIL* A7« mUS . be taken t0 understan ° the logical relationships 
between different components of benefits so that double-counting does 
not occur; or, If U occurs, 1t 1s clearly Identified by the analyst. 

A GENERAL TAXONOMY OF COSTS 

There are six different cost components Included 1n a benefit-cost 
model of an employment or training program. These cost categories 
include: l) program operating costs; 2) administrative costs at and 
above the SDA level; 3) opportunity costs to program 
part1c1pants--noted as Indirect costs below; 4) other costs associated 
with a rrtlclpant's attendance in the program treatment; 5) psychic 
costs to participants of undertaking the program treatment; and 6) 
program stipends or their Income transfer payments, such as UI benefits 
received during training. 

Operating and Administrative Costs . The cost of operating employment 
and training programs Includes both direct operating costs such as 
salaries for Instructors, materials and staff development costs and the 
cost of administering the program. None of these costs are counted as 
costs to program participants, as program participation 1s free. Both 
operating and administrative costs do, however, represent costs to 
non-part1c1pants and society, m Exhibit 3 these costs are represented 
as a zero to program participants and as a negative 1n the other two 
columns. (The negative sign 1s used to denote the presence of a cost 
Item. ) 

Opportunity Costs to Participants . Persons participating In 

government-sponsored employment and training programs may forego 
earnings opportunities they may have had otherwise had they not 
participated. The wages they would have earned while participating 1n 
the program are a cost to them of participating 1n the program. This 
opportunity cost to participants 1s not balanced by corresponding 
benefits to non-part1c1pants, and thus enters the social benefit-cost 
calculation as a cost to society, as well as to program participants. 

Another way to view this cost 1s that, from society's point of view, a 
person s decision to enter a subsidized employment training program may 
reduce the output the person might have produced 1f the person were 
employed at that time. This foregone output 1s a net cost to society; 
Its value 1s measured as foregone earnings. In Exhibit 3, these losses 
are characterized by minuses for participants and society and a zero 
for non-part1c1pants, who are unaffected by these opportunity costs. 

Since a comparison group exists 1n the data set to be used to measure 
post-program effects on total earnings, 1t 1s possible to estimate 
foregone earnings of participants while they are members of the JTPA 
program This can be done by Identifying the modal time period over 
which training (or any other program treatment) occurs. Indeed, you 
would ideally separate program participants by their type of program 
treatment In order to perform this analysis. Next, since there 1s a 
comparison group available, the before-Lax earnings of this comparison 
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group car. be estimated during this modal time period. These earnings 
must then be compared to any before-tax earnings that the program 
participant group under study may also have earned during the modal 
time period of program participation. The difference between the two 
estimates will represent the foregone before-tax earnings that should 
be counted as a social cost. After netting out any income taxes and 
social security taxes, the resulting difference 1n after-tax earnings 
should be counted as a cost to the average participant 1n the treatment 
cohort being studied. 

Increased Program Treatment Costs. Participants 1n employment and 
training programs may Incur costs that are associated with attending 
classes or receiving other treatments such as job search services. 
These Include such costs as transportation, child care, the extra value 
of meals bought away from home compared to home meals, any special 
clothing and equipment and expenses for materials and supplies 1f these 
are not supplied free to the participant. Some of these costs, such as 
the Incremental costs of meals purchased away from home, are difficult 
to measure and may represent only a small share of the total costs of 
the program treatment to the Individual and society at large As a 
practical matter due to evaluation cost and time constraints, one would 
conrlne the measurement effort to the largest and most easily measured 
cost items— usually those where a cash outlay is Involved and hence 
where they are easily Identifiable. In general, these costs are those 
reported on JTPA Annual Status Report and its underlying documentation. 

Psychic Costs of Participating 1n Training or Job Search Programs 
Anyone who has ever sweated out an examination or attempted to puzzle 
through a mathematical problem or the translation of a complex sentence 
in a foreign language is familiar with the psychic costs of learning 
So, too, 1s anyone who has had to search for a job. These types of 
costs accrue to program participants and to society but not to 
non-participants. They are inherently unmeasurable but are included 
here for the purpose of completeness. It would be tempting to argue 
that these psychic costs are balanced at the margin by psychic benefits 
that derive from a sense of accomplishment. But this is an empirical 
question and we have no way to measure directly these psychic benefits 
and psychic costs. The availability of psychometric scales qualifies 
this statement—changes 1n self-esteem can be indirectly measured But 
It is difficult to conceive of a psychometric scale that would measure 
the pain-cost and stress of training or job search and be commensurable 
with any Index measure of psychological benefit due to the program 
Hence, these costs and benefits are noted as important, but not 
measured . 



Stipends and Other Income Transfe r Payments . The possibility of paying 
stipends to individuals who are participating 1n the JTPA still 
mt 1 ts - Likewise, the Act as well as state laws allow the payments of 
Jf n f f1 i s t0 U1 el1 9lble Individuals, when properly certified, during 
partidpat on in the JTPA. These payments represent costs to the 
non-participant and benefits to the participant. Since they ar* 
transfer payments, they net out to a zero cost for society; the cost to 
the non-participant is cancelled by the benefit to the participant 
There Is no net additional drain on society's resources as a result of 
payment of these stipends to the program participant. 
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THE PRACTICAL PROBLEM OP COST ESTIMATION 

^t P blar 0 s JS tl,e eC «;t "Tn* V^f"*'*- ba ". d on the '"dividual or group 
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this nature In the currently available data are discussed below 

Mal«1? r Hl,!f"t S „ V\ %t \ F °I 1nstan «. the problems Imposed on cost 
. PriCe ""P"*"*""' of program Inputs Impart considerable 

:S w Sor3 

however 5 Hence' It ! h"' ^ 00 measur1n 9 m ^g1nal costs, 
]oln^ dl ^r^ « as the 

the f^cllitv Vo^, in 9 man ! 9 ? d by 3 common ^^ger. Some part of 
tS operVtV/n 0^:^^!^^^^ ^ SMt "' contribute to 

of 1o?nt e inn»t? ?n°V° r 3 subcontrac tor) has a considerable proportion 
whJe Kinf P ?ii ♦ Program, state reimbursement 1n cost Categories 

ti%hp J ?ni I PUt l u° CCUr w111 result 1n transfer payments or sub? Id el 
to the SDA since the marginal or additional cost to a jointly Produced 
output is zero. In ef fect, the SDA may receive payments for which 

?n 6 Samuel ^LV^ N / " cKean - " The «*• of Shadow Prices." 
in Samuel B Chase, Jr. editor. Problems 1n Publl r Expenditures 
Analysis. Washington, D.C.: The B rookings Institution. 1968 

di«u»Jd t bSi5l! p between tota1 ' average and marg1nal costs '« 
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little or no reciprocal services have been rendered. There 1s nothlnq 
wrong with subsidies, of course, but sound policy decisions should 
separate what 1s a subs1dy-an Income redistribution— from a true 
economic cost, wherein additional resources are consumed 1n the 
production process. 

This leads us directly to consideration of costs versus transfer 

S^Vc'tlnrfr 011 K a * pol1Cy s * and P° 1nt . tw ° benefits will follow from 
this distinction between cost and transfer payment. First, 1n future 
contract negotiations Involving employment and training programs, 
program administrators at the state level will be able to evaluate true 
economic costs more accurately. Second, previously unidentified 
subsidies In currently operating programs may be Identified, since no 
necessary service may be rendered In return for these subsidies they 
can be reduced or shifted among programs, 1f so desired. Unlike a cost 
reimbursement for the actual use of resources, the elimination of a 
subsidy or transfer payment need not necessarily negatively affect the 
objectives or desired output of the program. 7 

AN INTRODUCTION TO ECONOMIC COST THEORY 

A major aspect of this paper 1s to present consistent estimates of the 
average and marginal costs of employment and training programs based on 
Federal government expenditures and for Federal government plus any 
state, SDA or other agency or subcontractor expenditures. 

In addition, this paper will Indicate how to measure marginal cost, the 
extra cost of enrolling an additional participant Into a program 
Marginal costs are extremely Important 1n benefit-cost analysis 
because, when compared to the additional or marginal benefit received 
from a JTPA project, they Indicate the restive productivity of the 
project. The decision rule facing a program manager 1s to continue to 
SH?:? re , sources t0 a Particular project or activity as long as the 
additional or marginal benefit exceeds the additional or marginal 
cost. When the two are equal, total benefits from the project or 
activity are at a maximum and cannot be Increased by additional 
Investment in that specific project or activity. 

Cost Interrelationships. Economists employ various measures to show 
the relation of cost to output; among these are total , average , and 
marginal cost - (See Appendix A to this chapteFTFr a graphical 

6 A 2. 1 " d1cated n below ' a transfer to an Individual 1s a payment for 
which no equal, reciprocal service or benefit 1s offered, it 1s a 
gift, 1n effect. A transfer payment 1s sometimes called a subsidy. 

7 Consider the example of a surplus public school that has been 
converted to JTPA training facility. If the surplus school had nS 
other use 1n the community and could not be sold to someone 1n the 
private sector, then Its capital cost or rental cost to the JTPA 
program 1s zero; it has no alternative use. It 1s the existence of 

i We U ! e-_a fore 9° ne benefit-that defines cost. Any 
additional renovation, conversion, maintenance or operating costs 
should, of course, be attributed to the program. 
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depiction of these costs.) Total cost 1s the total outlay for a alven 
r«? U V< *H eraq H e ^ St 1 S t°tal cost divided by total output? marg na 
M of oXf^ t0 t0tal C ° St dUe t0 the Production' of 

The Measurement of n.itn.it. cost measures are expressed 1n terms of 
some measure of output. The output variable 1n this study 1s usually 
some variant of the number of persons trained 1n a productive actlv tj 
r, U n h 3S ^ 0rk ' S h ch00l c 0r the m1l1 tary, over a given time period^ It 
can be the number of persons subject to placement 1n a job. Briefly 
the problem Is this: The number of persons trained or subject to 
placement 1s only a proxy for the true output measure of an employment 

utn tr ca n o 1t 9 al Pr o 0 f 9r f * I" 15 J™ .° UtpUt me3SUre 15 the Increase In* t 
human capital of a worker due to the training or placement Drocess 
However, as a practical matter, this Increase 1n human capital cannot 
be measured directly 1n a physical sense but only Indirectly In a value 
sense-by how much a person's wage rate rises or earnings Increase A 

a r1se n in a9 hl/n t r e h Can H be V? , 1nCreaSe 1n 3 P-son's output due to 
cnnllVts I I T C3PUal < assum1n 9 the price of the output 1s 

Stout is con«*5 r nn e th l°" tput he or she P rod ""s (assuming that 
nn+nn; nL C0 J S ant) ' or both. Thus , the physical measure of 
output-one's Increase 1n human capital- becomes confused with the 

"'k?* "\ m ° f th3t ° utput - No one has * et successfully solved this 
Th'Ss l° f i 1dent1fy1n9 the 3Ctual chan9e 1n embod1ed human capital! 
S " r tVir 3 P T° Xy ' the number of per o sons twined at some average 
level of skill or educational prof 1c1ency .8 Suc h a proxy measure mav 
overstate or understate the true measure of human capital Increase Y 

Specification of Cost Fnnrtw costs of education and tralnlnq 
programs are analyzed using two^cost functions: a total cost YuVcUn l 
f n J an av erage cost function . The total cost func tlnn no r^77 
ltnZl\ cost'ofT, 3b0Ut ^Ml _cost. I.e., the e xtrTor 
Da?t?c?nSnt m* °/ l tra1n J n F one more trainee or placing ^nT~more 
KJt « th Mar ?. 1na ! cost 1s derived by computing the change 1n total 
fniLfni the / esul * ° f a un1t change 1n the number of persons trained 
(placed). As noted above, marginal cost estimates are useful 1n 
answering questions on whether to expand a given program or project 
from Us current size and 1n determining the efficient operating level 
*LS£ P T )9ram K°i r pro A ct - Als °. as noted above, using marginal costs 
S n he r Pr ? b } em ? f having to allocate joint costs among different 
types of related outputs, e.g.. to different training classes 1n the 
same building, assuming classroom training of all types 1s aggregated 
Into a single measure of output. 



8 



Numerous studies mainly of educational production functions, use 
measures of performance on standardized tests as proxies for the 
Increase In human capital. This can also be done In JTPA 
programs. However, the exact nature of the relationships between 
these proxies and the change or level of human capital 1s not well 
?uJrIw < S6 u S , ame J B ° wleS ' " Toward an Educational Production 
C^ui? ^ ? anSen ' EdU ° r - Education. Income and H.^n 

£ifilMl. Studies In Income and Wealth. No. 35. New York, NY: 
Columbia University Press. 
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™ e average cost function can also be used to obtain Information about 
the optimum scale of operation for a training program. The average 
cost function, 1f 1t has a U-shape as 1n Appendix Exhibit 4, permits 
inferences to be made about the output level that will result 1n a 
minimum average cost per person trained. One can Identify the 
]? a ^7 C0St po1nt of out P ut - The average cost function 1s derived by 
dividing the total cost function by total persons trained-total 
output. We do not recommend any attempt to directly estimate the 
average cost function econometrlcally. 

EXAMPLES OF COST ESTIMATION 

This section describes the available data on total costs, total output 
and certain variables that may affect the nature of total costs as 
total output 1s produced. It also presents both hand-fitted and 
econometr1cally-f 1tted total cost functions and discusses their 
Interpretation and limitations. 

An Empirical Total Cost Function . As Appendix A sets forth 1n some 
detail, the total cost function expresses the functional relationship 
between total cost (TC) and total output (X). The JTPA Annual Status 
Refiort 1s the source of data for estimating the total cost function. 
These data are limited, but can be used successfully. At this time 
possible complex relationships between total cost and total output 
cannot be Investigated because of sample size limitations. Looking 
ahead, 1n this case, the loss of Information 1s not great because 
limited experimentation with the data suggest that for the set of SDAs 
In Washington State, the total cost function 1s linear and marginal 
costs are therefore constant. 

We therefore specify a linear total cost function to analyze statewide 
SDA cost behavior: 

TC = a + bX + u 
Where 

TC = total federal costs 1n $100 per quarter; 
X = total output, expressed either as total termlnees or total 
terml nee/weeks 

a = the regression coefficient that estimates fixed costs; 

b = the regression coefficient that estimates marginal costs- and 

u = the error term. 

The unit of observation, because of the few number of observations 
available, 1s the SDA. Cost analysis 1s therefore done at the state 
level. Analysis could not be performed for a single SDA because there 
were too few quarters of data available on which to perform analysis 
and data were not available on a monthly basis. An SDA, however can 
perform econometric cost analysis as Its monthly observations on total 
cost and total output accumulate. Finally, the time period over which 
the cost and output data are relevant 1s 7/1/84 through 6/30/85. 

Two definitions of total output are discussed: total termlnees and 
total termlnee/weeks. A total cost function 1s fitted manually for 
total termlnees. Experiments with different forms Indicated that total 
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»h?« C ° S h l at3 ' The C0St data that are used as samples for analysis in 

W hlna on Thp re H,S aSed 0n „ data for SDAs 1n the State of 

Washington. The data are structured for reporting to the Federal 

'En M S? d ^mSE b l Sed -° n « t J l ; JTPA Annual strtl ETA 8580 

■fiduciary reports 3 P Y selected variables from these 

Strengths 

JUnfA 11 * 1 ! 0 " 4*° *5f ma1n Station discussed above there are several 
additional strengths and weaknesses to these data as reported A main 
strength of the data is that they are displayed ? .Vl-ar and 
SJ!SaS;? US H W3y - T . / he def1n1t 1ons of the variable^ a 
IJ/Jll^ ? rWard ' Tota J expenditures (but not total social costs) of 
Federa funds are shown for a given time period, as are total 
participants and total teVminees for the same time period. These three 
data items are the primary building blocks of any cost analysis. Total 
Federa expenditures can be expressed either as a function of total 
participants or as a function of total termlnees. Exhibits 4 and 5 and 
L 1 9 analysis express total costs as a function of total 
termlnees because reporting form ETA 8580 reports on the socio-economic 
characteristics of termlnees. These socio-economic characteristics can 

XlSirH^ 6 and .< eX l e , nt 0f Costs and therefore can be used to 

relat1 ° nsn1 P betwee " total costs and total output, 
ndeed, individual participant characteristics can be thought of as 
Inputs entering into the employment and training process. 

For instance, there are obvious instructional and resource implications 

•M» !h ! i, th K at , S ° me ° f the JTPA P ar t^1pants and termlnees have 

ted En 9 11sh Language Proficiency." Before they can be trained 
either their English language proficiency must be improved or the 
USES™?! mat A Mal l 3nd 1nstruct °r must present the training in the 
hSI %? a } S , n3t1 ? , t0n9Ue - E1tner c °nd1t1on will likely increase 
both fixed and variable costs, shifting the total cost functions up 
I.e., raising average and marginal costs. "Average weeks participated" 
at time of termination is another important variable. Clearly other 
tnings equal as average participation in weeks increases, total costs 
are certain to increase. 

If weeks in the program are related to some specific training 
structure then you can expect a more complex relation between weeks 
and total costs. Experimentation with the other socio-demographic 
^"^^ft^V" ^ 'How so me imp °rLt 

Weaknesses 

Although the above analysis would provide considerable useful 
Ih«I a * 0n ° n P rogram cost s, there are also certain weaknesses to 
tnese data. The most important weakness is that the data do not 
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display the number of termlnees by type of program treatment , while 
there are about two dozen different, but related, treatment approaches 
listed In the JTPA, as discussed 1n previous chapters, they can be 
condensed to a few main types: classroom training, on-the-job training 
and Its variants, and job search or job placement services and their 
variants.* Also, multiple treatments are possible for a given 
participant. a 

If total termlnees were disaggregated and enumerated by type of 
treatment, the following would be achieved: 

1. One could estimate a much more accurate relationship between total 
costs and total outputs since the termlnees receiving different 
treatments represent different Inputs and, therefore, potentially 
different cost structures. Incorporating this additional 
Information Into the estimation of the total cost function may 
Improve the statistical reliability of the marginal cost estimates. 

2. Even though total costs are aggregated Into a single number 
marginal costs for each type of treatment can be directly estimated 
by expressing total costs as a function of each of the termlnee 
treatments considered simultaneously. The equation below expresses 
this possibility: 

TC = a 0 + bi X] + b 2 X 2 + b 3 X 3 + b 4 X 4 +C] T + d 1 + u 
where 

XI = classroom training only termlnees; 
X2 = on-the-job training termlnees; 
X3 = job placement only termlnees; 
X4 = termlnees with multiple treatments 
T = Average weeks at termination; 

= a vector of socio-economic variables such as limited English 
language proficiency, sex or high school grade, or equivalent 
and above. 

ao. bj, b 2 , b 3 , b 4 , C] , and d 1 = coefficients tc be 

estimated; and, 

u = the error term. 

The outcomes of this estimation will be estimates of marginal rnst f nr 
each type of treatment even 1f the SPA or state has aggregated these 
total costs such tha t they can't be separated 1n an accounting sense . 
This 1s a major advantage of using econometric techniques to estimate 
program costs. 

A final problem with these data 1s that they do not Include any 
estimates of costs contributed by the state to the operation of these 

9 Some program operators have been used to considering job placement 
as a program treatment. More correctly, 1t 1s an outcome : one 
either Is placed 1n a job or one Isn't. The costs of job 
placement however, are Inputs and these Inputs represent the 
proper measure of program treatment. 
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K the sda wi Mf necessaril y important for the estimation of costs 
actual , 6V V ,f ?* 3re concerned with only SDA performance), any 
actual or Imputed costs contributed by the state to the operation of 

£El P r°J*Tl Sh ° Uld be ^ ded t0 the quarterl - v and annual program yea 

Sit tS« T,t SU , re * St l te l6Vel and ful1 economic costs - " appears 

[social) cost Hat 6 ]" th f h 0nly , maJ ° r P ° tent1al om1ss1on fr ™ the 

participants foregone before-tax earnings of 

oalment; Im^ nll'T ™^ tenance Spends are allowable In JTPA, such 
payments will not be a large component In reported federal data 

2m?o 4*„ Pa ™ Cip l nt !u Can also receive ""employment insurance payments 
while in JTPA and these must be identified when they occur in the 
accounting record. Both types of expenditures must be omitted from any 
cost analysis at the social level. y 

ESTIMATING COSTS AT THE SDA LEVEL 

?nrm EX !!J b l t * < Sh ° WS ; C ? st °A ta colle cted at the SDA level are In the 
«Mp A ™ of observations on costs and outputs over t1«t--tXI 
series data. The data available on SDAs In unpublished form at the 
nw!n nPr "VL b *> at best ' 0" ar terly data. As Exhibit 4 Tho^iT 

XllIJwIJ? I S 1St ° ry < ° f JTPA ' th1S currentl y does not provide enough 
observations to do meaningful cost analysis at the SDA level. Exhibit 
4 shows how one should take the raw data and organize the data into a 
display that allows a straightforward description of the cost and 
output components of the SDA. With these data, average costs by 
quarter can be estimated by simple division of total costs by total 
Rnl a Al°" per quarter or by total termination-weeks per quarter. 
But, with the existing shortage of observations ,thi s 1s as far as we 

^?c 9 °; T /°J al C u St funct 1° ns and marginal costs cannot be estimated at 
this time (December 1985). 

dJETnV ™! n°^ S «»1nta1n program data on a monthly basis. Monthly 
dill ,h J / b °r Ve M^ 1d Pr0v1de about 15 observations within the 

Jrnnr.J f\ f ?J « J?"? 1 * *-- enou 9 n for cost analysis of the SDA's 
program If it is settled down into its long run course of operation and 
if there are no significant recording errors In the data. Under this 
arrangement a total cost function and marginal cost can be estimated. 
A procedure that has been tested before and which works 1s described 
below It was designed to analyze the cost and performance experience 
(DCC) 10 6r Commun1ty Conference Economic Readjustment Program 

Formulation of — a Dynamic Total Cost Function for t h* 
Typical SDA . 11 

Due to the nature of an SDA's program there 1s reason to assume that 
there Is a good deal of inertia In the data; that is, adjustment to any 
new level of activity Is not made instantaneously and unused capacity 
may tend to persist. Concern over "lumpy" start-up costs is an 
expression of this phenomenon. 



10 
11 



DCC is now an SDA. 

This model is taken directly from Kullk, et al., 1982. 
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EXHIBIT 4 



TTTI C TTA rMDHI 1 MCMT AMH PfiCT HATA 


Or SERVICE 


DELIVERY 


AREA "X". 


TIME SERIES DATA. 


Variable 




Accounting Period 






10/1/83 


10/1/83 


7/1/84 


7/1/84 


7/1/84 




3/31/84 


6/30/84 


9/30/84 3 


12/31/84 


3/31/85 




/ m \ 

(1) 


(21 


\ 0 J 




(5) 


Total Adults 












Total Participants 


258 


326 


271 


456 


702 


Total Terminations 


73 


253 


61 


195 


385 


Male 


36 


121 


30 


87 


166 


Female 


37 


132 


31 


108 


219 


Less than high school graduate 


9 


46 


4 


25 


60 


High school graduate or more 


64 


207 


57 


170 


325 


Single household head with 












dependent children 


19 


75 


18 


54 


119 


White 


55 




Oc 


1 7/1 


345 


Black 


3 


r 
O 


0 


4 


9 


Other 


15 


40 


9 


17 


31 


Limited English 


13 


22 


4 


7 


14 


Handicapped 


7 


34 


6 


23 


59 


UI claimant 


8 


46 


9 


28 


55 


Unemployed 


66 


230 


48 


145 


277 


Average weeks at termination 


10 


12 


22 


16 


16 


Total Federal Program Costs 












in $1 DD ( rumii 1 a 1 1 up ^ 




$9,130 


$2,428 


$6,138 


$10,083 


Difference: (2)-(l); (3)-(2); etc. 




$3,384 


$2,428 


$3,710 


$3,945 


Total Termination - Weeks 












(total terminations x average 












weeks at termination) 


730 


876 


1,342 


2,768 


3,040 


Average Cost/Terminee - Week 




157 


$181 


173 


130 


Average Cost/Terminee 




1,880 


$3,980 


2,769 


2,076 



Source: Job Training Partnership Act Annual Status Report; Various quarterly 

reports as noted by dates above for SDA "X" in the State of Washington. 

Note: a 7/1/84 is the beginning of the accounting period. It appears that a large 

quantity of fixed or "start-up" costs were expended during this quarter. 



nnfm.V^n ?k SU *« 1-P ll C ? S J function 1 " which costs are related to 
output in the relationship below: 

(1) C = C Q + C ] (q) (This 1s the same as TC = a + bx.) 
Where, 

C = cost 
q = output 

This formulation 1s relevant when the economic activity is at a steady 
state and the future can be predicted accurately. In our case, the 
production and delivery of program services entails a learning process 
by program managers. Our data will be monthly and during the early 
months of the program the administrators can be assumed to be moving 
towards some "steady state". Our problem 1s to formulate a cost 
function that reflects (a) the learning process In the short run, and 
(b) the tendency toward a steady state (a stable operating rate) 1n the 
long run In this sense the cost function is dynamic- -it reflects 
adjustment over time to a stable operating rate. 

We can write the cost function as: 

(2) C t = o< + q * t long run cost function 

(3) C. = c* + /? q? short run cost function 



Where 



q* t = expected output 1n period t 
q t = equilibrium output 

In the case of the short run cost function, the SDA providing services 
expects some level of output and 1t prepares to meet that requirement 
with its planned staff and other resources. The costs, therefore are 
related to the expected level of output, rather than the actual output. 

We assume a learning process by the SDA 1n the form of adaptive 
expectations. The term below represents the rate of adaptation. The 
SDA can be modeled as revising Its future expectations on the basis of 
past experience. Specifically: 



or 



(4) Q* t - Q* t _l = <«t -«*t-1> 
(4') q* t = \q t ♦ (l- A )q* t r 

Combining (4 1 ) and (3) we get: 

(5) C t = W +/U q +/?(1- \ )q*t_i 

35 
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Lagging (3) one period, multiplying through by (1-X). and 
the result from (5), we get: 



(6) C t = (l-X)C tl +A*+/^q 



This 1s the autoregressWe form of the cost function which we 
estimate. A note on the estimation of this cost function 1s 1n order. 
Estimation of an autoregressWe function by ordinary least squares will 
result 1n biased and Inconsistent estimates of the parameters Yet 
given the small number of observations that are likely available (15 
months) and the transitional nature of the program during the first 
phase of operation, the use of more sophisticated techniques to 
overcome bias and Inconsistency 1s ruled out at this time. As more 
observations— months of data— become available, 1t will be possible to 
adjust for the problems 1n the data more effectively 1f additional 
variables exist to adjust for the bias. 
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ESTIMATION OF PROGRAM COSTS AT TK2 STATE LEVEL 

Much more Information on costs and outputs exists at the state level 
since the state can augment Its data by drawing Information from all of 
the SDAs 1n Its jurisdiction. It 1s not limited to the experience over 
time of only one SDA. 

In the State of Washington there are ten SDAs. Data for a full year of 
operation are available on each SDA as shown 1n Exhibit 5. These data 
are known as cross-section data. They are a cross-cut of experience 
throughout the state at a given point of time. Since the data reflect 
a period well after JTPA program start-up, one can assume that the data 
reflect relatively stable operating patterns. In any case, start-up 
costs are now spread over a year or more. As with the data for a 
single SDA, one can array the variables of Interest 1n an easy to read 
table and make comparisons across SDAs. 

If you are willing to assume that, 1n general, the SDAs use similar 
approaches to managing their programs, e.g., for certain types of 
training, they all tend to use proprietary vocational schools, while 
for other types of training, they all tend to use public vocational 
technical scnools, then 1t 1s possible to average the experience of the 
SDAs across the state. (Any aggregation of data makes assumptions of 
this nature.) Normally, this averaging will be done with a 
mult1-var1ate regression model as has been discussed above. 

Total Cost — Based on Total Termlnees . However, to explain the 

principles of the analysis, we have shown an approximate estimate of 
the total cost function, and the marginal cost and average cost 
functions based on 1t, by fitting a total cost function by 
Inspection— that 1s, estimating the nature of the total cost relation 
and fitting a linear total cost function with a straight-edge. 1 2 

Early efforts at multiple regression and correlation often 
described this technique. See Ezeklal and Fox, 1959. 
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TITLE IM ENROLLMENT AND COST DATA FOR SELECTED SERVICE DEL I VeV AREAS STATEWIDE. CROSS-SECTION DATA. REPORT PERIOD 7/1/84 - 6/30/85. 



Variable 


SPA 1 


SPA 2 


SPA 3 


SDA 4 


SDA 5 


SDA 6 




cnfl O 
JUH 0 




SDA 10 
























Total Participants 


471 


807 


3,241 


932 


936 


451 


750 


UlO 


oUo 


WO 


Total Terminations 


413 


650 


2,232 


717 


729 


332 


TO 


dfl7 


79G 

(ft 


ICC 

3dd 


Male 


191 


329 


1,030 


301 


449 


185 




loo 


T71 

3/1 


177 


Female 


222 


321 


1,202 


416 


280 


147 




9CG 


41)8 


108 


Less than high school graduate 


85 


117 


433 


104 


171 

til A 


JO 




OA 


183 




More than high school graduate 


328 


533 


1,799 


613 


557 


276 


dlQ 


TCI 


c/o 
S4Z 


295 


Single household head with dependent children 


142 


187 


792 


239 


182 


77 


228 


lid 


one 


lib 


White 


351 


586 


1,136 


356 


634 


275 




ODD 


cno 


34U 


Black 


19 


6 


446 


85 


12 


12 


27 

LI 




oc 
to 


0 


oo Other 


43 


58 


650 


276 


83 


45 


77 


70 


l:Jo 


26 


LimHea tngnsn 


2 


8 


355 


232 


66 


7 


14 


13 


18 


8 


Ma no i cdppeo 


65 


27 


385 


81 


50 


55 


67 


24 


55 


35 


111 riatmant 


57 


111 


250 


60 


113 


26 


96 


73 


134 


67 


Unemoloved 


3U/ 


CO/I 

bo4 


1,922 


634 


671 


308 


444 


341 


700 


312 


Average weeks at termination 


14 


17 


19 


16 


12 


18 


22 


18 


19 


20 


Total Federal program costs in $100 


$7,765 


$12,885 


$37,232 


$14,684 


$12,124 


$12,018 


$7,844 


$10,23-' 


$12,301 


$5,854 


Total termination - weeks 






















(total terminations x average weeks at termination) 


5,782 


11,050 


42,408 


11,472 


3,748 


5,976 


11,704 


8,046 


13,775 


7,300 


Average cost/terminee • week 


$134 


$117 


$88 


$128 


$139 


$202 


$67 


$127 


$87 


$80 


Average cost/terminee 


$1,880 


$1,982 


$1,668 


$2,048 


$1,663 


$3,620 


$1,474 


$2,289 


$1,697 


$1,604 



Source: JTPA Annual Status Report for the program year ending June 30, 1985. Various SOAs. 



Exhibit 6 shows first a scatter plot of the annual data for each SDA. 
(SDA#3 was not used since it was an extreme measure.) The solid line 
1s an approximation of the total cost function fitted by hand" If this 
Is a good approximation of the total cost function for the JTPA program 
in the state of Washington for the 1985 program year, then we see two 
Important observations. 

First, total costs are linearly related to total output. There are no 
significant economies or diseconomies of scale of operation within the 
range of the data we have available. The linear total cost function 
reveals that marginal costs are constant and they run In the 
neighborhood of $1,750 per program termlnee given that all forms of 
program services have been averaged together.' 3 

However, It Is much more accurate to fit such a cost function by means 
of a regression model. Given the few number of observations, this can 
easily be done on a personal computer such as the IBM-PC. The 
estimated total cost function based on an ordinary least squares (OLS) 
regression program has the following form: 

TC = $412,625 + $1,193 X 
(284,094) (501) 

The adjusted coefficient of determination (R 2 ) is e H jal to .37. The 
data are Interpreted as follows. 

First, on the average, for the nine SDAs that were Included In the 
analysis ■« fixed costs were $412,625 per SDA per year. This estimate 
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It Is Important to remind ourselves that as the number of data 
observations Increase In quantity It will be possible to 
disaggregate the termlnees by type of program treatment and 
thereby estimate marginal costs for each type of program treatment. 

SDA 3 was omitted from both the hand plot and the regression 
estimate because It was an extreme value--an outlier. There is no 
hard and fast rule for editing data to remove outliers or other 
anomalies. Given sufficient observations, SDA 3 could have been 
left In the model and. a variable been Included to adjust for that 
phenomenon that accounted for Its large size. With so few degrees 
of freedom however, It was best to omit SDA 3. 

However, a regression model that Included SDA 3 gave the following 
results: 

TC = $223,986 + $1 ,548 
(119,398) (135) 

with an adjusted R 2 of .94 or 94 percent. 

The high R 2 suggests a much better fit leaving SDA 3 In the 
sample. However, another way of working at these results Is that 
SDA 3 dominates the estimation so that the regression Is really 
being fit between two points: SDA 3 and the average of the other 
nine SDAs. Under this situation the mathematics of regression 
analysis will give you a biased estimate. 
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Exhibit 6 



COST ESTIMATION AT THE STATE LEVEL: THE SIMPLE RELATIONSHIP BETWEEN 
TOTAL FEDERAL COSTS (EXPENDITURES) AND TOTAL TERMINEES 



Total 
Expendi- 
tures 
in 

$100,000 



17 



16 



15 



14 



13 



12 



11 



10- 



Total Cost Fitted by Inspection 
TC = f(x) 

MC = 175,000 / 100 
= $1,750 




'10 



Total Cost Fitted by 
Regression 

TC = $412,625 + $1,193 
MC = $1,193 



300 400 500 600 700 800 900 nW 

Total Terminations / Program Year 
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1s statistically significant at about a 10 percent level of 
significance. Next, marginal costs equals $1,193; that 1s, for the 
program year 1n question, across all nine SDAs, the additional cost of 
producing one more program termlnee was about $1,200. This estimate 1s 
also statistically significant at conventional levels of significance. 
Finally, we note that total termlnees explain 37 percent of the 
variation 1n total costs for the program year. Given the few number of 
observations, this 1s a reasonable result. With more observations 
additional variables could be added that would explain more of the 
variations 1n total cost and yield a higher coefficient of 
determination (R^). Breaking this data set Into monthly or quarterly 
observations by SDA, for Instance, would Improve our ability to 
estimate the total cost function, though a somewhat different model 
specification would be required to account for the time series nature 
of the data. Standard economic procedures exist to do this as are 
discussed, 1n part, above, for the dynamic SDA regression model. 



BASIC METHODS AND ISSUES IN BENEFIT-COST ANALYSIS 

Social Investment Criteria. There are three criteria that can be used 
to measure the net effectiveness of a social Investment: the Internal 
rate of return, the net present value of money benefits and the 
benefit-cost ratio. Exhibit 7 shows the mathematical expression of 
these criteria. Two Important points should be reviewed concerning the 
Implementation of these criteria. First, as discussed previously, 
marginal benefits and costs must be employed with these criteria. This 
1s because the Incremental benefits of any social Investment must be 
equated with the Incremental costs that created them. As long 

as the Incremental benefit exceeds the Incremental cost, one should 
continue to devote resources to the social -Investment. Once the 
Incremental benefit equals the Incremental cost that created 1t, no 
further gain 1n total benefits can be made by further Inflow of 
resources to the soda! Investment program. 

Second, recalling Exhibits 1 and 3, one should not mix social concepts 
of benefit (cost) with participant or non-part1c1pant measures of cost 
(benefit). To do so 1s the economic equivalent of comparing apples 
with oranges. 

As a final assessment to this discussion, 1t 1s necessary to put the 
nature of any program evaluation 1n Its proper context. Paraphrasing 
Gramllch (1981), 1t 1s sufficient to consider a social Investment 
program a success 1f the benefits yielded by 1t represent a net 
Improvement— the present value of benefits exceeds the present value of 
costs. It 1s not necessary--or even practically possible— that a 
program be a panacea. Thus, a job placement strategy that costs an SDA 
$200 per participant need only to yield $200 1n the present value of 
benefits (at the socially required discount rate) 1n order to be 
socially justified. It does not have to guarantee the worker against 
unemployment for the rest of the worker's Hf e . Of course, 1f two 
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net benefits equalize, whichever comes first. exhausted or the 
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Exhibit 7 
INVESTMENT CRITERIA 



Criterion 



Formula 



Internal rate of return 



T B t 
E — ' 
t=0 (1+1)* 



T c t 

E , 

t=0 (TTrp" 



= 0 



Net present value 



T B t T C t 

E 7 — - E ^_ > o 

t=0 (l+r)t t=0 (l+r) 1 



Benefit-cost ratio 




Ct 



> i 



B 
C 
t 
r 
T 
1 



benefit per unit of time 
cost per unit of time 
the time unit 

the Interest rate used 1n discounting 
total number of time units 

the discount rate which gives a net present value of 0 
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APPENDIX A 

EXAMPLES OF TOTAL, AVERAGE AND MARGINAL COST FUNCTIONS 
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APPENDIX A 



This appendix sets forth three common types of relationships between 
the output of a government-sponsored employment and training program 
and the Inputs or resources used to produce that output. 

Economic theory and empirical studies narrow the range of possible 
equation forms for cost functions. Once a total cost form Is 
specified, the form of the average cost function derived from It Is also 
specified. 

The four equations listed below represent the general form of the 
alternative total cost equations discussed In this appendix. 

(1) TC = a + bX + ul 

(2) TC = a + bX *■ cX 2 + u2 

(3) TC = a + bX * cX 2 + dX 3 + u3 

(4) Log TC ■ log a + b log X + log u4 

Schematic diagrams of the first three equations are provided 
The variables In each definition are defined as follows: 

TC = total cost; 

X = total output; 

a, b, c, and d represent partial regression coefficients; and 

ul, u2, u3, u4 = error terms to formally complete each model. 1 

Equation (1) Is the linear formulation of the total cost equation. The 
term a represents the amount of fixed costs In the total cost function 
The expression bX says that for any change in the number of persons 
trained, X, total cost will change by some constant multiple relative 
to the change in persons trained, bX. Therefore, if the number of 
trainees changes by one trainee— one output un1t--then total cost will 
change by a constant, b, relative to that change of one unit 
regardless of the level of total output. Regression analysis Is used to 

1 1n J the x P rev1ous chapters, the error term encompasses those 

Independent variables which may be relevant to the model but 
which, for whatever reason, are omitted. It Is assumed that the 
omission of these variables does not bias the estimates of the 
effects of those variables that are explicitly Included In the 
model . 
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statistically estimate a linear total cost function. The regression 
coefficient, I.e., b, for the trainee variable 1s the estimate of 
marginal cost 1n the case of the linear total cost function. 

Equation (2) Is the quadratic form of the basic total cost 
relationship. It 1s quadratic because total cost Is hypothesized to be 
related to the total amount of output plus the square of that total 
output. In simpler terms, this 1s just one variation of a non-linear 
relationship between total output and total Input. The marginal cost 
of the program, I.e., b ♦ 2cX, Is again found by using the same method 
of differentiating. Now, however, In order to determine marginal cost 
with the quadratic form, once the derivative 1s found, a specific level 
of output must be substituted Into the equation. In other words, the 
marginal or additional cost In this equation form Is dependent upon the 
level of output, I.e., marginal cost Is not a constant quantity 
relative to output levels, but changes as the total amount of output 
changes per unit of time. 

This difference between constant marginal cost and a changing relation 
for marginal cost reflects two different hypotheses concerning the 
nature of the relationship between total cost and total output. In the 
linear case, It Is hypothesized that total cost bears some constant 
relationship to output over a range of output ratos. Thus, If output 
goes up (or down), costs will Increase (decrease) by some constant 
multiple of output. No Increasing or diminishing returns to Inputs 
exist as the output expands. Each new additional Input unit Is just as 
productive as the last. However, In production processes, Including 
education or training production processes, this may not always be the 
case. As output 1s expanded In the short run, the productivity of 
successive Input units may rise or fall. The Intuitive reasoning 
behind this changing productivity, and thus decreasing or Increasing 
marginal cost, Is that some Inputs are fixed In quantity so that with 
the addition of more units of one kind of Input, either more efficient 
.use of the total set of Inputs occurs and productivity Increases or 
crowding and Imbalances occur and thus productivity decreases. The 
quadratic equation form and other non-l1near forms specify changing 
marginal costs because marginal costs vary with output levels. 

Equation (3) 1s the cubic form of the total cost function. It Is so 
called because one of the Independent variables In equation (3) Is a 
cubic term. Marginal cost calculations can be made using the same 
method described for equation (2). Equation (3) allows for diminishing 
or Increasing returns to an Input used In an employment and training 
program, as does equation (2). 

Equation (4) 1s the logarithmic formulation of the basic total cost 
relationship. The total cost equation In logarithmic form, apart from 
the fact that 1t may fit a given set of data better than the other 
three forms, can be used to make direct Inferences about the elasticity 
of costs relative to output. Elasticity 1n this sense Is defined as 
the percentage change In total cost divided by the percentage change In 
total output. Elasticity values are expressed In percentage terms and 
thus enable proportional cost response comparisons to be made among 
various production processes such as four different employment and 
training program treatments. The elasticity value Is directly 
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- ■ : Appendix Exhibit- 1 

ALTERNATIVE COST CURVES: THE LINEAR FORM 




TC = a + bX 



1 i 



Total Trainees /Unit of Time 




Total Trainees/Unit of Time 



Note: 



The scales on the axes of__ the Total and Average and Marginal Costs 
are not the same. i a a 
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Appendix Exhibit 2 
BASIC DEFINITIONS FOR APPENDIX FIGURES 1, 2, AND 3 

o Total Cost (TC) equals Total Fixed Cos ts (TFC)— Administrative and 
Start-up Costs— plus Total Variable Costs (TVC)— Operating Costs other 
than Administrative Costs. 
TC = TFC + TVC 

o Average Costs (AC) equals Total Costs divided by Total Output (X) 
AC = TC 

r 

o Marginal Costs (MC) equals the change (A) in Total Costs as Total Output 
changes by one additional unit or increment. 

MC =&TC with respect to X. It is the slope of the TC function, 
a = TFC 

b, c and d = Coefficients that measure the slope of the TC function. 

NOTE: All of the above values are expressed in terms of a common time 
period, e.g., total costs per week and total trainee completors per week. 
The length of the time period is not important. 
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ouW t6 L by l he r * 9ression coefficient for the logarithm of total 
output— the number of persons trained. 

JL*, he f. 1 ; 8 * 1 /, 1 ,*^ 18 flre f ter than one > then the percentage change in 
™w ?n be 9r ! 3ter than the Percentage change In output as 

output increases, m other words, average costs will rise as a result 
of an Increase In the total number of persons trained. If the 
trill™ i«V 1e " °™> .-then a change In the total number of 

trainees will result in a smaller percentage change In total costs and 

ra'nae 96 SV^i W \ ]1 fal1 35 ° UtpUt iS expanded "ver some 

constant as t^?u ~ ~ / then 3Verage cost w111 remain 

JS2 „il I 6 u Ult ,° f an 1ncrease ^ total persons trained because 
the percentage change In total cost equals the percentage change In 
total output. From cost elasticity measures one can Infer the optimum 
rate of operation, that Is, the point where average cost 1s at a 
enterprise 3 " 1mp ° rtant cons1der at1on In the efficient management of any 

Four alternative average cost functions are applicable In this study 
The average cost functions are companion forms of analysis to the total 
cost functions. The average cost functions discussed In this chapter 
are formulated as follows: 

(5) AC = a/X + b + u5 

(6) AC = a/X «■ b + cX + u6 

(7) AC = a/X * b + cX + dX2 + u7 

(8) Log AC = log a + (b - 1) log X + log u8 
The variables used are defined as follows: 

AC = total cost divided by total output 
X = total output 

a, b, c, and d = partial regression coefficients 

u5, u6, u7, and u8 = error terms to formally complete each model. 

^! \ h A* 1 the \ verage A° St T? t10ns (5) « < 5 >« < 7 >' and (8) are derived 
from total cost equations (1), ( 2 ), (3), and (4), respectively, by 
dividing each of the total cost equations by the total output variable, 
" owev ® r ' equation (8) Is not typically estimated because equation 
(4) directly answers the question of Increasing or decreasing total and 
average costs as output changes. 

Technological factors can work to create Increasing or decreasing total 
and average costs because technological levels In employment and 
training programs may be reflected In training situations that have a 
discrete output capacity. As a result, training capacity Is "lumpy » 
An SDA may be faced with the situation of choosing a site that can 
optimally train 100 trainees per month; that Is, for this location and 
size of plant, average cost 1s lowest at a rate of 100 trainees per 
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month. However the projected training center may enroll at a more 
than optimum rate. Thus, actual average costs will be greater than 
average cost would be at the optimum rate of output. Average costs thS 
can be reduced If the rate of output drops back toward 100 per month 

2Jur r hS:.r ,0 5 a g1ven ?, pac1 ? fi other th1ngs equa1 ' avera 9 e cost i* 

usually higher for a small rate of output because fixed costs are 
spread over a low rate of output. As the rate of output Increases 
average cost decreases since average fixed cost per unit of output 
deceases; a point 1s usually reached In the rate of output, however 

resu r its a Tn ra a 9e rUr n re3CheS * l^*™' Expans1 ° n be * ond th 1s Point 
results m a rise 1n average cost because additional units of variable 

inputs GCme effic1ent « t^y are forced to work with the f?xed 
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GRAPHICAL EXAMPLES 

A linear total cost function 1s shown 1n Appendix Exhibit 1 Total 
costs change by a constant amount as output Increases. Each additional 
unit of output, e.g., a trainee who completes a training program, costs 
IV ame T J° p r od,,ce as eacn Previous trainee termlnee In the SOA or 
state. This gives us a flat marginal cost curve because the extra cost 
of producing an extra trainee termlnee remains the same for each 
additional trainee. Finally, the average cost of producing a trainee 
termlnee divines over the full range of program operation because the 
nlnlrJ 0 ? T * U rt-up co l ts f°r running a particular program or 
program treatment are spread over larger and larger amounts of total 
output while marginal cost stays constant. Eventually, the average 
cost effectively equals the marginal cost as a/X approaches zero. 

fL a n i° ted * ab u Ve I . 0n , e can est1mate the average cost curve by dividing 
total costs by total output, as follows: 

TC=AC=a*bX=a+b 
X XXX 

m]^?n/i 1Xe V°fJ S a u re f1xed ' they d0 not cnan 9 e as ^tput changes, 
marginal cost-the change In tctal costs as total output changes by one 
additional unit--1s equal to b; or, 

HC = b = output ^ K respect t0 a one unit change in total 

!«J?? d 2il C C l St fun ^1on 1s shown 1n Appendix Exhibit 2. Here, total 
costs change by an Incre^sina amount as total output Increases The 
equation for this total cost measure adds an additional variable X 2 
the square of total output, which yields the curvilinear shape of the 
function. Dividing the total cost function through by total output at 
an* given leval of total cost, yields average costs: 

K = AC = a t bX «. cX 2 = a ♦ b ♦ cX. 
x X X X X 

Note here that as X increases in size the value of a/X approaches zero, 
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Appendix Exhibit 3 
ALTERNATIVE COST CURVES: THE QUADRATIC FORM 



Total 
Cost 




TC = a + bX + cX< 



TFC = a 



Total Trainees/Unit of Time 



Averaqe 

and 
Marginal 

Cost 




MC = b + 2cX 

AC = £ + b + cX 
X 



Total Trainees/Unit of Time 



Note: The scales on the axes of the Total and Averages and MarainaJ 
Cost diagrams are not the same. 
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while the value of cX becomes very large. As cX Increases 1n size 1t 
approaches the value of 2cX 1n the marginal cost function. The value 
of b 1s constant in both the marginal and the average cost function, so 
that at high values of output marginal and average costs tend to 
converge. 

Finally, Appendix Exhibit 4 displays th* cubic total cost function. 
When divided by total outpit, the cubic cost function yields the 
conventional U-shape to the average cost function. 

TC = AC = a+hX + cX 2 + d_X 3 = a+ b + cX + dX 2 . 
X X X X X X 

As total output increases, the term a/X causes average cost to 
decrear*. However, at the same time, the terms cX and dX 2 cause 
average cost to Increase. cX and dX 2 domlrnte a/X as X Increases and 
average cos' Increases as total output Increases. As mav be clear at 
this point, the average cost and the marginal t curves' are related. 
In this cost relationship, at iome point the marginal cost of producing 
an additional trainee termlnee Increases with each additional trainee 
termlnee who 1s added to total cost. This Increase Is due to the 
general phenomenon known as the law of diminishing returns. This 
phenomenon, 1n turn, 1s due to the relationship between fixed Inputs, 
such as a single teacher or a classroom of a given sire and the number 
of variable 1nputs--most Importantly the participants to be 
tra1ned--relat1ve to that teacher or classroom along with any equipment 
the class may require. As you add more stude.its to the fixed classroom 
or require a single teacher to Instruct more students, as 1n, say, a 
vocational-technical class 1n screw machine operation, the Incremental 
amount of knowledge Imparted per unit of teacher Input will begin to 
drop. There are a variety of reasons for this: The room gets more and 
more crowded, the teacher's span of control over students 1s 
attenuated, Insufficient attention may be given to each student--all 
relevant questions may not be answerable, there 1s Insufficient 
machinery for each student to acquire the optimal practical experience, 
and so on. 

In Appendix Exhibit 4 the law of diminishing returns 1s fundamentally 
responsible for the curvilinear shape. However, the relationship 
between the fixed and variable Inputs 1s such that at first there are 
Increasing returns- -Incremental output Increases as an additional unit 
of a variable Input 1s added to the set of fixed Inputs. This can be 
due to some complementary relationship between the fixed and variable 
Inputs; for Instance, 1n Instructing students 1n screw machine 
operation, the most efficient class size may be 20 students, cue to the 
technical relationship between the teacher, the students, the required 
mix between formal and practical Instruction and so on. Thus, the 
effectiveness of Instruction Increases up to 20 students, then becomes 
constant, and, as the class expands further beyond the optimal size of 
?C students, Instructional effectiveness decreases . Hence, the 
particular shape of this total cost function— total cost first 
increases at a decreasing rate, then becomes constant briefly, then 
switches to increasing at an increasing rate. This 1s expressed in the 
U-shape of the marginal cost function and the average cost function. 
The equations that express the value of margin*! and average cost, at 
any level of total output, are shown on the figure. Their general 
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'and 

'Marginal 
Cost 




MC = b + 2cX + 3dX 2 



0 Total Trainees/Unit of Ti 



Notes: The scales on the axes of the Total and Average and Marginal Cost- 
curves are not the same. 
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derivation and Interpretation 1s similar to that In Appendix Exhibits 1 
and 2. 



We should note that an SOA or subcontractor that Is operating an 
employment and training program could conceivably have a total cost 
function that would take on any one of these three forms (as well as 
other forms that are not shown here), depending on the nature of the 
relationship between outputs and Inputs 1n the programs being offered. 
If one were to break down a program by type of treatment or services 
delivered, It 1s possible that each service or treatment could have a 
different total cost function. In fact, even If the basic form Is the 
same--say, the total cost function Is linear for each of several types 
of training, you could expect the average and marginal costs to differ 
for each type of training. Empirical analysis of costs has shown this 
to be the case. 
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U.S. DEPARTMENT OF LABOR 
Employment and Training Administration 

JTPA 

ANNUAL STATUS REPORT 



OMB Approval No. 12 
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PREFACE 



The evaluation guide presented in Volume V for studying the impact of 
JTPA on the recipients of its services utilizes a sophisticated non- 
experimental research strategy. Before making this choice, an assess- 
ment of a range of methodologies was made. Approaches used in 
evaluating previous employment and training programs were reviewed. 
JTPA s unique program characteristics and context were considered. 
The climate in which the program was designed and is being implemented 
was given attention. The level of interest in the evaluation mandate 
in the legislation, and the evaluation capability of states and local 
JTPA administrative organizations were judged. 

Despite these careful considerations, the choice not to use an 
experimental approach— that is, the approach considered to conform 
most closely to scientific principles and methods-was a complex and 
difficult one. Why? The following discussion addresses this question 
by exploring a perpetual methodological debate in policy research 



64 



CONTENTS 

INTRODUCTION 

CONTEXT OF THE DEBATE 

Defining the Controversy 

The Significance of the Controversy for States and SDAs 

SCIENTIFIC TRADEOFFS IN THE DEBATE 
The Issue of Bias 

Biases Common to Experimental and Nonexperimental Designs 

Major Differences Between Quasi-Experimental and Experimental 
Designs 

Tradeoffs Between Quasi-Experimental and Experimental Designs 

ORGANIZATIONAL AND POLITICAL TRADEOFFS IN THE DEBATE 
Organizational Issues 
Political Issues 

CONCLUSIONS 

REFERENCES 



SUPPLEMENTS 

A Definition of Concepts 

B. Some Sources and Kinds of Bias 

C. A Research Design Continuum 



ERIC 



05 



INTRODUCTION 



A distinguishing characteristic of the human species is intolerance of 
uncertainty and the unending search for explanations which reduce it. 
This search has been a persistent feature of all societies irrespect- 
ive of the nature of their development. The complexity of human needs 
and abilities supports this pursuit of information about "reality". 

The form this search takes may be sacred or secular. The explanations 
may range from myths to carefully gathered empirical evidence. Policy 
research, at its best, occupies a position at one end of this 
continuum, representing a systematic scientific approach for un- 
covering "the truth" about the effects of social policies and the 
programs designed to carry them out. But the most competent 
scientists can only approximate the truth. The best they can do is to 
state educated probabilities that things "are" as they are able to 
perceive them through the analysis of unbiased information. And not 
even the most rigorous scientific methods are perfect in removing all 
the potential biases which stand between the evaluator and what is 
true. 

This conclusion is important to a discussion of the longstanding 
debate in policy research between the use of experimental vs. non- 
experimental research designs in seeking the truth about social 
programs. It is often assumed, in this eternal argument, that 
certainty is possible if only the proper methodology were used 
However, the issues are far more complex than that. 

This paper probes the nature of this polarized debate to provide a 
context in which the reader can better understand and appreciate the 
choices made in the GUIDE FOR NET IMPACT EVALUATIONS, which utilizes a 
rigorous nonexperimental approach. In studying this controversy and 
its implications for the evaluation of JTPA impact at the state and 
local level, it is important to discuss the following: the background 
of the debate; the continuum of research approaches typically used in 
making assessments of social programs and the kinds singled out in the 
debate; and the scientific, organizational and political tradeoffs 
involved in making choices. 



CONTEXT OF THE DEBATE 



The greatest heat in the debate has been generated in skirmishes 
between policy researchers advocating for the use of experiments, and 
those arguing that rigorous nonexperimental designs are more 
appropriate in most policy settings. A lack of consensus among these 
groups with respect to the definition of concepts central to the 
debate complicates this argument over which research appro 2 ch actually 
brings us closer to the truth, and therefore provides a better basis 
for improving policies and programs. 
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But there is nothing new about the controversy. This debate has been 
raging for at least six decades, beginning with the Thorndike era in 
psychology in the 1920s. Questions about the use of controlled 
experiments have continued into the 1980s and are likely to remain. 
However, there is growing recognition that no one ideal scientific 
solution is right for all circumstances. Researchers of both 
persuasions, in the physical as well as the social sciences, find "the 
truth 11 almost as elusive now as when the controversy began. What 
continues to unite these professional peers, however, is the 
conviction that the application of scientific method to the study of 
how effective social interventions may be is far superior to more 
subjective judgments of their value. This common cause is one 
explanation for the growth of applied research, and its selection for 
use in the social policy arena in this era of "social accountability". 

So what is this debate all about, and why is it a significant issue 
for states and SDAs in evaluating programs under the Job Training 
Partnership Act? 



Defining the Controversy 

The debate turns on the primary goals of scientific inquiry, which 
influence the choice of issues and methods. These goals involve the 
endproducts of research: the information acquired and its inter- 
pretation. Accuracy and utility are the critical ingredients of this 
information. More specifically, the important characteristics of 
scientific information are reliability , validity , and general i za - 
bijjty. In applied reseach an additional characteristic is the 
usefulness of the i nf ormati on for pol i cy purposes . Beyond this, 
another goal is the rep! icabil ity of the research itself, and its 
conclusions, in other relevant contexts and settings. 1/ 

To achieve these goals, principles and methods are applied which 
express logical steps in scientific method. We go to the trouble of 
conforming to this hard-won historical approach because it brings us 
closer to "reality" than any other form of assessment. In more 
technical terms, it helps the evaluator reduce errors and biases 
associated with the way an evaluation is carried out, and the 
conclusions arrived at on the basis of the evaluation. In general, 
the chronology of this evaluation process is as follows: 

Selection of issues which are feasible and useful to study, 
based on what is already known about these issues and the 
purpose of the research. 



1/ A glossary is provided in Supplement A of this volume which may 
help clarify the meaning of some of the key terms used here in 
describing research objectives and methods. 
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Correct definition and measurement of the factors considered 
most important in shaping or otherwise affecting these 



issues 



Development of an appropriate research design guiding the 
selection of the individuals, groups or organizations to be 
studied— and the collection, analysis and interpretation of 
reliable information which can provide valid inferences 
about the influence of the factors selected for study. 

Selection of reliable methods for collecting information 
about the influence of these factors. 

Selection of appropriate methods for analyzing the 
information gathered. 

Cautious interpretation of the results of the analysis, 
including what insights and conclusions can be drawn, and 
their generalizability to the intended subjects of the 
research and to the body of knowledge about the issues 
studied . 



Repetition of the research in alternative contexts and 
settings, to see if similar inferences can be obtained fe- 
ttle same kinds of research subjects. 

The major scientific objective of program evaluation is to guard 
against error and bias in seeking the truth about the influence of the 
program s interventions on a particular set of desired outcomes for 
participants-apart from the influence of other factors which may also 
influence program outcomes. 

It is now well understood that explaining the results of attempts to 
change people's attitudes and behavior is a monumental task. But it 
is less well understood that this becomes more difficult if the inter- 
?mn^J2nt th . at * re ex P^ted to change people are incremental, and other 
important influences in their lives are not or cannot be measured. 
Since most social programs involve mainly short-term limited efforts 
to change those who experience social problems, the evaluated s 
^ P ° nS H b H U K SS : e P res8nt a Oenuine challenge. This challenge is 
compounded by the mcreasing public demand for accountability in 
tax-funded programs, and the growing public appreciation of the 
ttnn^ m TH nS T f val1able for Judging the effectiveness of such 
TZT/f ^hese developments are occurring in the context of severely 
reduced funding for social programs and their evaluation. 

Ukes Dlac e fr "^Tn,t Within r h V Ch ^ assessment of Program impact now 
takes place h« mo st important scientific problem is developing ways 
to make valid estimates of the effects of public services and other 
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forms of social intervention. The debate over what kind of approach 
should betaken in accomplishing this task must therefore address the 
problem of error and bias. 2/ 

Significance of the Debate for States and SDAs 

Before discussing further the scientific aspects of the debate, it is 
important to outline its significance for JTPA. One of the most 
significant evaluation mandates for states is the emphasis in the 
legislation on "the return on investment 11 . To comply with this 
mandate, net impact studies must be carried out. The research designs 
used to implement such studies effect the accuracy with which the 
return on investment can be estimated. If JTPA outcomes for the 
intended population are as the Act directs, and these outcomes can be 
attributed to the influence of the program's interventions, the 
benefit warrants the investment in policy terms. However, if the net 
effect is not positive the policy implications are quite different. 
The Act essentially places a heavy burden on the evaluator to use 
those research approaches which best specify the return with a high 
level of confidence that the estimated impacts are as close to the 
true effects as possible. 

Therefore the continuing debate about what constitutes the best 
research design affects the credibility of the evaluation choices made 
at the state and local level. But the state and SDA contexts in which 
such credibility is judged are significantly different. And these 
environments are distinctly different from the national context of 
JTPA evaluation. 

States are vulnerable to strong, well organized state-wide public 
interest grrup pressures that argue against withholding services from 
one group of eligible clients while providing them to others in on- 
going programs, which ;s a requirement of experimental designs. Also, 
research designs cannot be allowed to interfere with client, employer 
or union prerogatives in relation to such programs. Such pressures 
have a substantial although sometimes indirect effect on the operating 
decisions of state agencies, including evaluation choices. SDAs are 
highly dependent on the continuing good will and support of local 
elected and appointed officials, employer and labor organizations, and 
social service coalitions, which can influence public, professional 
and client perceptions of locally managed government programs. 



2/ Although the research definitions of "error" and "bias" differ, 
the term "bias" is used throughout this paper in the nontechnical 
sense, to refer to both errors and biases in the evaluation 
process. 



Despite pressures placed on the federal agencies by the congress and 
well financed and highly organized political, economic and public 
interest lobbies, the research divisions of these agencies are not 
usually the primary target of lobbying activities. Although public 
support for large-scale experimentation at the national level has 
always been weak, there has been growing tolerance for limited field 
experiments which affect only a small subset of any stay's or local 
service provider's clients, and occur only 1 in a few selected state and 
local sites. This is well illustrated by tr.e ability of Ihe U.S. 
Department of Labor to recommend a small num? ar of JTPA field experi- 
ments, while simultaneously analyzing data on a representative sample 
of all JTPA clients using a nonexperimental approach. The political 
and organi zat i onal compromi ses at the federal 1 ev.*l are based on 
different tradeoffs. 

One must conclude that the research design debate has < uite different 
parameters at the state and local level, where even limited experi- 
mentation inevitably involves a significant proportion of the clients 
of a social program. Those engaging in the controversy often fail to 
recognize that there are multiple tradeoffs to consider at each level, 
which require compromise. 



SCI ENTIFIC TRADEOFFS IN THE DEBATE 



The consideration of scientific tradeoffs takes precedence in making 
evaluation choices, even though pol itic^l and organizational issues 
act as a sieve through which high priority research approaches are 
sifted for their broader feasibility and utility. Therefore major 
emphasis in this paper is placed on scientific compromises. The 
following issues are addressed: the issue of error and bias in 
evaluation resear h; the continuum of research designs whicr. respond 
to this issue; common soirees of error and bias; msjor differences 
between experimental and quasi-experiment ?u designs ; and the advan- 
tages and disadvantages of both approaches. 

The Issue of Bias 

It is cJifficult to avoid subjectivity in determining whether policies 
and pro^rajis work as intended. The sources and varieties of potential 
bias whiefv^cientific method seoks to control are numerous. 3/ These 
biases can be introduced at any stage in the evaluation process: the 
conceptualization of issues to be studied; their measurement the 
sampling of the individuals to be studied; the selection of a research 
debign for collecting and analyzing information about them; data 
collection; data analysis; and the interpretation of evaluation 
results. 



3/ An array of examples is provided in Supplement B. 
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The following are some examples of the kinds of biases that must be 
dealt with in program evaluations. A primary task of the evaluator is 
to avoid as many as possible through the way in which the evaluation 
is planned and implemented, and to try to compensate for those that 
cannot be controlled by these means. 

For example, the outcomes of clients involved in an evaluation may be 
influenced simply by participating in a study. The additional 
attention alone may affect their attitudes toward the program and the 
way they utilize its services. Such attention is an unmeasured non - 
program intervention which may bias an estimation of the program 1 s own 
effects. Or the evaluation may focus on the less relevant issues, 
missing the critical ones. Perhaps the relationship between single 
hea d of household and earnings is a key one in understanding the 
effect of the program on welfare status, but this is not studied. Or 
it may be that the most important relationships have been properly 
identified but the variables are poorly measured. For instance, in 
defining program interventions such as job search assistance , or 
program outcomes such as employment , too few indices may have been 
used, or the measures selected may not have been consistently defined 
across all clients in the study. Evaluation results can also be 
biased because they are not representative of the program 1 s target 
group due to i nadequate sampl i ng , in whi ch case they may under- 
represent important groups such as women or youth. 

Difficult to control are biases associated with clients 1 own self- 
selection into JTPA, and the initial selection of clients by program 
staff. Not all eligible clients will apply. When applicants exceed 
program resources staff will naturally enroll those eligible 
applicants they judge to be most in need, those best able to benefit 
from the program, or those who will most likely help them comply witn 
performance standards. In sorne cases these clients may be those most 
capable of achieving program goals without the program's assistance. 

Sometimes clients 1 outcomes are compared with those of a group of 
individuals who an> not sufficiently similar. If the comparison 
group's prior labor market experiences are significantly different 
than the JTPA group, for example, this will bias the results. Or the 
essential information to be collected may not have been reliably 
gathered because of the failure of data gatherers to adhere system- 
atically to scientific interviewing principles and procedures. Or the 
analysis of information may utilize inappropriate methods. Also, 
differences in program histories, contexts and environments may be 
ignored in ths analysis. Once results are obtained, their interpret- 
ation may be biased. Conclusions may be drawn based on the mere 
existence of a "statistically significant 11 difference in outcomes, 
neglecting the strength of that difference. Or the results of one 
evaluation may be inappropriately compared with those of a sub- 
stantially different study, such as comparing net impact studies of 
JTPA with previous net impact studies of CETA. 

Minimizing such biases is not easy. The researcher must be astute in 
identifying those that will impede the discovery of the truth in a 
particular evaluation, and develop ways to reduce them. 
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In this respect, the choice of a research design is critical. It is 
therefore important to recognize the wider spectrum of choice in terms 
of research designs, to place the debate into better perspective. 

A research design is a plan which guides the gathering of reliable, 
valid information on the appropriate subjects of the evaluation, and 
directs the analysis of this information to determine causal or other 
kinds of relationships among the factors considered most important in 
understanding program implementation or impact. There are a number of 
alternative designs from which to choose. 4/ The traditional 
continuum distinguishes between experimental and nonexperimental 
designs, and nonexperimental designs are frequently differentiated as 
exploratory, descriptive and quasi-experimental based on their 
differing research purposes and their degree of rigor in reducing 
bias. 

The usual purpose of exploratory designs is to formulate new ideas and 
dovelop and refine research questions for more rigorous study. De- 
scriptive designs are intended to investigate associations or cor- 
relations among major explanatory influences and the outcomes of 
interest, short of attributing cause and effect. Quasi-experimental 
designs are expected to identify cause and effect relationships 
between interventions and outcomes, in the absence of experimental 
conditions and with the help of statistical controls. The purpose of 
experimental designs is to establish cause and effect, by both 
randomizing the intervention to the subjects of the evaluation and 
applying statistical controls. 

It would be incorrect to assume that all designs other than the most 
rigorous are inappropriate or lack utility in policy research. Each 
type of design makes its own contribution to useful information about 
a social program. However, in net impact evaluations, the rigorous 
end of the continuum is involved due to the emphasis on determining 
whether or not the program "caused" the effects. 

Because of the search for causal relationships and the intransigence 
of bias, the controversy over quasi-experimental and experimental 
designs has gradually narrowed to those alternatives considered most 
effective in studying cause-effect relationships in terms of reducing 
numerous sources and kinds of bias. That is, the debate is now 
largely confined to experimental designs vs. the most rigorous quasi- 
experimental designs. 

Biases Co mmon To Experimental and Rigorous Quasi-Experimental Designs . 

There are some common barriers in making valid estimates of program 
impacts. Again no step in the research process is exempt. The best 
developed and implemented experimental and quasi-experimental designs 
cannot control for all the potential biases that interfere with un- 
veiling the truth. 



4/ See Supplement C for a more detailed overview of this continuum. 
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For example, an evaluator cannot rule out, with either design, those 
biases due to poor conceptualization of the issues to be studied or 
their inadequate measurement . If a classic theory useful in studying 
employment and training programs in the past is inappropriately 
applied to a contemporary program which is quite different, important 
factors may remain unmeasured and unstudied. Neither can an evaluator 
prevent through either kind of design the effects the personality or 
behavior of the evaluator may have on clients 1 outcomes. 

Also, these designs cannot control for the effects of unanticipated 
external events which occur to clients, the program or the program's 
environment during the period of the evaluation, such as unexpected 
family crises, changes in program leadership or natural disasters. 
They cannot control for the impact of unknown or unmeasured program 
implementation policies or practices. Both designs leave uncontrolled 
those biases which are due to the combined effect of a number of 
influences on outcomes, such as the combined influence of a client's 
age, sex and education on postprogram earnings, unless these effects 
have been anticipated and trssse characteristics measured. 

But it is important to acknowledge that with both designs evaluators 
can attempt to compensate statistically for common sources of bias by 
identifying and measuring as many of the factors which may bias 
results as possible, and adjusting for them in their analyses. 



Major Differences Between Experimental and Quasi-Experimental Designs 

Despite common problems, there are important differences between the 
two kinds of designs being debated. The differences are based on the 
extent to which these designs meet the scientific test in terms of 
reliability, validity, general izabi 1 ity and repl icabi 1 ity. Within 
that test, the degree to which they accurately establish cause and 
effect is at stake. To explain these differences we need to begin 
with the ideal experiment and work our way toward the realities policy 
research must confront. 



The Ideal Experiment 

The goal of the "ideal" experiment is to achieve maximum control over 
influences which prevent the evaluator from discerning the truth about 
the phenomenon being studied. The truth in net impact studies is the 
discovery of which influences contribute to specific program effects 
in a logical cause and effect series of relationships. In program 
eval uat i on a 1 arge number of i nf 1 uences may cause the outcomes 
measured. The evaluator seeks to eliminate factors which compete with 
the program's treatments in explaining those outcomes. 

In the ideal experiment many of the influences other than the service 
interventions are controlled. That is, they are eliminated as 
significant influences on program outcomes. Using certain procedures 
the evaluator can isolate the separate effects of a number of treat- 
ments operating simultaneously or sequential ly to affect a set of 
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outcomes. Through the random assignment of individuals to treated and 
nontreated groups, the experimenter can rule out the effect on out- 
comes of certain influences that are unknown, or known but unmeasured, 
thereby reducing bias. 

The influences that compete with the program's treatments to explain 
outcomes are of several kinds: (1) factors which may be confused with 
or mistaken for the program's interventions—such as a major change in 
job opportunities in the local labor market; (2) factors which can be 
controlled by random assignment—such as measured characteristics of 
clients prior to their receiving services, for example their attitudes 
toward government assistance; and (3) factors which can be controlled 
through the use of statistical procedures— such as the selection of 
clients with particular characteristics to receive certain services, 
or the duration of the services assigned to them. The objective is to 
control for as many competing explanations as possible. 

In the ideal experiment most competing explanations are control led 
through randomization procedures alone, or randomization accompanied 
by statistical techniques. A well designed experiment permits the 
evaluator to make a minimum number of appropriate assumptions about 
such influences, exercise optimum control, and make causal inferences 
with greater confidence. In the evaluation of program impact, this 
means that the evaluator can come very close to determining the true 
effects of the program's interventions. 

The key feature of experimental designs is randomi zation . This 
process sorts research subjects randomly into an "experimental" group 
and a "control" group prior to the introduction of the "experimental 
treatment". Translating this principle for JTPA, it involves 
randomizing eligible enrol lees into a group which receives JTPA 
services and one from which these services are withheld, before the 
treated group receives any services. 

In the ideal experimant the research design may also involve matching 
prior to randomization . This process gives the evaluator precise 
control over particular influences which are known or thought to 
affect outcomes. Subsequent randomization controls for many of the 
influences that have not been anticipated. 

As an example of matching, assume the experimenter wants to study the 
differences between the outcomes of male and female JTPA enrol lees, 
and has good reason to believe that age and ethnicity strongly affect 
those outcomes. The evaluator also believes that attitudes toward 
work will exert a substantia 1 influence on their employment, earnings 
and welfare outcomes. Age and ethnicity can be more accurately 
measured than attitudes, so controlling for their effects poses 
different problems. Therefore individuals in the two groups can first 
be matched on the basis of age and ethnicity, which can be assumed to 
control for these two influences precisely. Subsequent randomization 
of each of these matched groups Into those receiving the program's 
services and those not, can be assumed to cancel out most of the 
effects of other factors operating prior to program oarticipat -ion-- 



such as the attitudes of these individuals (work motivation, for 
example), and the impact of unanticipated events (such as economic 
recession) for which controls are not available. 

Whereas randomization relies on chance to cancel out the effects of 
competing influences, matching gives the evaluator more direct control 
over certain characteristics, Used together, matching and randomizat- 
ion allow the researcher to more closely approximate equivalency 
between the two groups being compared. Differences in the outcomes of 
these two groups can then be taken seriously in determining a 
program's net impact. 5/ 

An additional element of the ideal experiment is the measurement of 
tte experimental and control groups with respect to critical variables 
both before and after the introduction of the program intervention to 
the experim^al group. In complying with this principle, the 
evaluator must assume that the measurement process is not itself 
affecting outcomes. Given that it is not, preprogram measurement can 
insure a baseline against which postprogram follow-up information can 
be compared. 

But despite matching, randomization, pre/postprogram measurement and 
the use of statistical controls, the ideal experiment is not 
infallible. There are many kinds of in teractive effects occurring 
between the program's interventions and competing explanations, which 
may jointly affect outcomes, These joint effects are often more 
complicated than simply adding together the separate effects of each 
influence. For example, the joint effects of age, sex and the program 
treatment on outcomes may be different from the separate effects of 
each of these influences. Even the most ricorous experimental design 
enables the valuator to study only a limited number of these inter- 
active effe'is, Also, the evador's assumption that treated and 
nontreated groups are equivalent may hold true only for the average 
effects Oi a program's interventions, or its effects on the "typical" 
client, rid only within the laws of probability— that is, the larger 
the sizt of the experimental and control groups, the more confidence 
one ct have in their equivalency. 

So, \y as we might, certain uncontrolled influences may remain which 
can obscure or confuse our effort to measure the true effects of 
p-,t r $ interventions. This simply reveals the natural limitations of 
t>; ,'ijman species and the complexity of reality, Nevertheless, the 
■m xperiment h the closest we can caw to estimating the true 
n>sck of the yy#m designed to carry out important social 



5/ If the experimenter must work with snail samples, matching prior 
to randomization is essential, But this requires the evaluator 
to accurately identify ahead of time those characteristics of the 
study group likely to be most strongly related to their outcomes. 
However, if the evaluator can use large samples, or if a large 
number of variables can be studied, randomization alone is an 
effective form of control. 
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Less Than Ideal Experiments 

In actual practice few experiments turn out to be ideal in all 
respects. As the debate has been narrowed to judgments about rigorous 
quasi-experimental vs. experimental approaches, there has been a 
tendency to assume that all quasi-experimental designs are alike, and 
all experimental approaches the same. In practice, the continuum of 
designs that fall within each of these two categories can vary 
considerably in terms of their conformance to or departure from the 
ideal experiment. 

For political, organizational and economic reasons, ideal experimental 
design principles and procedures cannot always be adhered to con- 
sistently in experiments involving ongoing programs, Neither can a 
meticulously designed quasi-experimental approach always be 
consistently implemented in real-life program environments. A flawed 
experiment, just as a seriously compromised quasi-experimental 
approach, can yield inaccurate information. If there is reason to 
believe that a field experiment's random assignment procedures will be 
violated, for example, a well planned and implemented rigorous 
quasi-experiment is preferable, 



Rigorous Quasi-Experimental Designs 

It is evident that the evaluation of net impact is more complicated 
when prior randomization is not feasible or practical, The evaluator 
must make additional assumptions which may be less plausible, and 
which may pose a greater potential for bias, The inability to 
randomize, and therefore to control for most other explanations of 
outcomes, is the primary problem confronted in using nonexpertental 
designs, But this problem is most troublesome when the evaluator 
desires to make causal inferences about a phenomenon which involves 
multiple interrelated causal variables, and complex relationships 
batween these interventions and their effects. This is the reality 
faced in studying social propjs such as JTPA. 

The major task under these circumstances is to identify as many of the 
competing explanatory factors as possible, measure them and control 
for them through statistical means. A range of optional designs has 
been developed which addresses ;iie challenges to reliability, validity 
and general izabil ity presented by an absence of randomization, some of 
which are ingenious adaptations of experimental designs, In these 
approaches, the statistical frameworks and analysis strategies of 
experimental designs are used, but not randomization. An example is 
the design recommended in this set of volumes for studying net impact, 
It involves pre/post measurement, a carefully constructed comparison 
group, and the use of statistical methods for identifying and 
adjusting for biases. 
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Tradeoffs Between Quas i-Experimental and Experimental Designs 

?Li n « ^ a . r } ? f P° 1it1cs . research designs involve compromises 
between the ideal and the possible. These compromises force us to 
consider tradeoffs with respect to three significant attributes of 
research: its realism, its scientific rigor , and its repre - 
sentativeness. 6/ Realism is a characteristic of how welTthe 
evaluator can identify, select and measure the influences and effects 
that occur in actual programs in practical situations. Rigor is a 
characteristic of how well the evaluator can control for bias 
Representativeness is a characteristic of how well the evaluator can 
select the study sample so that the results of the evaluation can be 
confidently generalized to the intended target population of a 
program. Severe compromises must often be made among these three 
qual ities. 

In general, the primary tradeoff in using well developed and carefully 
implemented experimental designs is greater rigor. With respect to 
well planned rigorous quasi-experimental approaches, it is greater 
realism. Systematic descriptive studies usually afford greater 
representativeness. Those designs that best combine these positive 
tradeoffs produce the most replicable evaluations. 



Advantages and Disadvantages of Quasi-Experimental Designs 

Rigor: As indicated previously, the major concern in using this kind 
of design to accurately determine the nature of cause and effect 
relationships is the lack of full control over other explanatory 
variables which may be confounded with the program's treatments. 

Because political, organizational and often ethical constraints 
discourage the treatment of ongoing programs as experiments, bias 
estimation and adjustment methods must do double-duty. Even so 
competing influences cannot be as efficiently controlled as in a well 
conceived experiment. Therefore it is imperative that the evaluator 
be aware of specific variables the research design fails to control 
This alerts the researcher both to .competing explanations and to 
potential biases which must be adjusted for through alternative 
methods . 

In constructing a comparison group, the evaluator must use care 
caution and imaginativeness in locating the most comparable 
individuals and the most reliable sources of essential information if 
bias is to be reduced. Statistical techniques can resolve certain 



6/ In his article on "Representation, Randomization, and Control" 
Leslie Kish introduce these terms in discussing design issues' 
The discussion here reflects some of the positions taken in this 
article. 
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control problems by allowing the evaluator to adjust for a large 
number of competing explanations, although assumptions must be made 
about how accurate the results of these adjustments may be. But 
identifying and compensating for all potential influences is beyond 
the function or capacity of statistical methods. However, it must 
again be understood that controlling for all intervening variables is 
a goal which can only be approached in any research. 

Real ism : One of the advantages of a rigorous quasi -experimental 
design is real ism. The eval uator has greater freedom than with 
experimental designs to identify, measure and study the broader range 
of variables and relationships which may more accurately mirror the 
true complexity of the program. For example, the evaluator can study 
the complexity of the service treatments as they are assigned by staff 
and experienced by clients in actual practice. However, such 
flexibility is limited by sample size. 

Representativeness : If quasi -experimental designs are careful ly 
developed there is also the opportunity to achieve greater repre- 
sentativeness than in experimental studies, in terms of revealing the 
true variation occurring within typical program populations across 
diverse state and local sites—even though this advantage is again 
limited by available sample size. At the state arid local level where 
experimentation is less acceptable, for instance, an experiment 1 s 
study population is frequently limited to that portion of the target 
group for which an acceptable political or organizational rationale 
can be provided for randomi zing services. The results of such 
experiments are not likely to adequately represent the target group of 
the program. 

In summary, rigorous quasi-experimental designs are generally weaker 
on control and stronger on real ism and representati vesness than 
experimental designs. They provide more f lexibi 1 ity in resolving 
conceptual and measurement problems , even though they leave certain 
research design problems only partially resolved. Because their study 
populations can better represent a program 1 s intended target group, 
their findings tend to be more general i zable than those involving an 
experimental design, even though it may be easier to replicate an 
experiment because of its economy of scale and scope. 



Advantages and Disadvantages of Experimental Designs 

Ri^or- Experiments are clearly stronger on control , since both 
randomization and statistical controls are available to reduce bias. 
Therefore experiments offer a greater 1 i kel i hood of i so 1 ati ng the 
effects of a program's interventions. 

Realism : Sometimes experiments are poorly conceived in the sense that 
they may be based on theories which suggest inappropriate explanatory 
frameworks, given the issues to be studied. The wrong variables and 
relationships may be emphasized, or the less important rather than the 
more significant ones. Although this is a problem in all research, 
experiments are particularly vulnerable because fewer manipulations 



• be made and fewer variables studied. As a practical requirement, 
... evaluator must simplify the reality of the program in order to 
< ercise maximum control. Pragmatic and ethical constraints may also 
place limits on the scope of experimental research. The evaluator 
must therefore be as correct as possible in identifying and defining 
the most critical variables and relationships to be studied. Also, 
because experimental designs must of necessity Halt the variety of 
treatments studied, the variation among the explanatory variables is 
often sparser than in the real world. 

Represent ativeness : Experiments are most often carried out in a small 
number of nonrandomly selected sites across the country. These ex- 
perimental sites tend to involve a population that is less diverse 
than the overall target population of a program. Such sites 
frequently have to be limited to those willing to randomize. They 
often over or underrepresent certain subgroups within the larger 
program population. Also, for political or organizational reasons 
experiments may have to confine the study group to a subset of the 
preferred population, such as to client volunteers, clients provided 
an incentive for participation, those willing to waive their legal 
right to privacy, or some other atypical subgroup. This will produce 
nonrepresentative study samples to varying degrees. 

In summary, experimental designs are generally stronger on control and 
weaker on realism and representativeness than rigorous quasi - 
experimental designs. They are less flexible in terms of resolving 
conceptualization and measurement problems. It is difficult to design 
experiments which represent the larger trrget population of a program. 
In addition, contriving the desired realism of a natural program 
setting is not at feasible in experiments. Therefore, in many cases 
the findings are less general izable than those yielded by rigorous 
quasi -experimental studies. 

In conclusion, it is important to recognize that this treatment of 
some of the tradeoffs between experimental and rigorous non- 
experimental approaches in evaluating social programs such as JTPA 
assumes that in practice most experiments can only approximate the 
ideal" experiment, and that even the most rigorous quasi-experimental 
studies will vary in the extent to which they can compensate for the 
absence of randomization. The main purpose of this brief overview of 
strengths and weaknesses will be well served if the reader has become 
increasingly convinced that the debate is more complicated than it is 
often presented and will continue to lack a definitive winner. 



ORGANIZATIONAL AND POLITICAL TRADEOFFS IN THE DEBAT E 



It is not enough to talk only about the scientific tradeoff? in this 
lively debate. Organizational and political risks and benefits attach 
to one or another scientific approach, and can have a majv vsoact on 
research choices. Just as certain research goals and qualities shape 
scientific decisions, broad acceptability and immediate utility for 
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policy, planning and service delivery purposes mold organizational and 
political decisions about whether to initiate an evaluation and what 
type of evaluation to perform. 

In many program evaluation situations, particularly in ongoing 
programs as compared with pilot and demonstration projects, experi- 
ments are simply not feasible. This is not unique to social programs. 
It occurs in the physical and biological sciences as well. In 
organizations, as in politics, practical significance means more than 
statistical significance. While evaluators have devoted themselves 
mainly to untangling the scientific tradeoffs, program practitioners 
and those in politics have preferred to play audience to this Olympic 
spectacle. But researchers, politicans and program operators alike 
are now keenly aware of the significant role organizational and 
political factors play in the debate, and the necessity of considering 
the characteristics of organizations and the nature of the political 
arena in tailoring evaluation decisions to pragmatic contexts. The 
most basic supports for making scientific assessments and using their 
results depend on this accommodation. 



Political Tradeoffs 

The larger environment of evaluation is political: the world of 
elected and appointed officials, special interest groups, and the 
powerful individuals, groups and community coalitions that support 
them. Experimental research can be justified to the American 
public— and therefore to these constituencies—better than before. 
The public is more sophisticated about evaluation research, and more 
interested in being given empirical evidence that there is a 
satisfactory return on their investment. The intrusiveness of 
experiments, such as dictating a particular form of client selection 
and service assignment, can be supported as an efficient way to test a 
new concept or approach before substantial public' money is allocated 
to it. This is especially true if an experiment has a low political 
profile because only a small proportion of a program's population will 
be affected, and only a few sites will be involved, This justificat- 
ion may also satisfy the political agendas of state training councils 
and PICs, who in turn are responsive to pressures generated in the 
larger political environment. 

Despite the public's greater knowledge and approval of scientific 
options in teeing innovations, politicians often feel they cannot 
rationalize a withholding of services from part of the eligible 
population of established public programs. Clients themselves play a 
role here. A high threshold of client resentment about service 
assignments which occur irrespective of a program's intended appraisal 
practices, or the impression of client exploitation for research 
purposes, will jeopardize a program's funding and the sponsoring 
agency s organizational safety. Also, as a prime political forum, the 
media are unusually sensitive to inequities. In an open society 
inequalities of service provision and possible civil liberties 
violations are easily identified and exposed. The most visible taraet 
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service" « 1 E * e " SUpport n 9 the of "those waiting for 

IlthiS r1.k "^ r ans + °" S^P, n . q^i-experimental studies is not 
service c L • ^i* P ub1 ' c 2es th * existence of incomplete 

fSi^^7?ow«T ,tw attentlon from act1ve client advocac * and 

t!I!2 fc r % h n0t - the °2 1y P° litical Problems that must be addressed. 
JoaTs vis-a ^ S s SU %° f *** Cons ; stenc y of a Program's goals, and its 
perverselv .ffprtpH h S meanS n f ° r - ach . ievin 9 them. Evaluation is 
KmEIi affect * d b - v P r °9 ram legislation that results from difficult 

h ^"Tr 5 "/ 01111 " 1 be1iefs define what social needs are 
to be reduced through the use of available resources. Political 
rationales therefore underly conceptions of what a social problem is 
and is oi, and direct the choices made among a range of oroqram 
alternatives for resolving a problem. These rationales Jnd directives 
are not always internally consistent. If there are competing 

^]s?^io n ^ v Ch h mUSt , be TJ ded b y.P° litical compromise, the resElt ng 
legislation may be rife with inconsistencies. 

Nevertheless, the evaluator is obligated to make judgments about the 
tf)l% i whlch P r °9 rams m eet their legislative intent efficiently and 

direct v ^nJ.V ^ XamPle i' StUdi6S ° f Pr09ram implementation must 
directly address the goals and means issue. However, what some 
political actors may most want in an evaluation is a confirmation of 
ZL ° r - t'H- ° f the Com P etin 9 Positions which spawned the 
h T ?" d6 I n lne 3 dl ' fferent P ur P° s e for the evaluation, 
and create disparate public expectations for the findings. In the 
' + eacn "nstituency may perceive different advantages and dis- 
advantages to be associated with any evaluation, or with experimental 
vs quasi-experimental studies, depending on what they view as most 
likely to produce findings favorable to a particular set of interests 
or political philosophy. When this happens, the luster of the 

SitSa^sss;,. 01, the debate may 9ive way to the briiiiance ° f <* 



Organizational Tradeoffs 

!nJ 1 l iC Vl fe< : tS or 9 a ni"tions. But evaluation decisions pose risks 
and benefits to organizations as well. These are largely a function 

th. sens n !n r Crat ? ti0n ''- 0f lar9e SCale orianizations and 
.nH f l? V J ty ° f 9° v erriment agencies to pressures from their f under s 
and immediate environments. Some of the characteristics of these 
bureaucracies support and son* hinder a full consideration of the 

t bat n e J% SC1 ii nt l flC , t . ra . de ° ffS - Eva l ua t^g Programs is a risky as well 
alS ? beneficial exercise for them. How programs le to be 
evaluated orten increases the organizational threats more than it 
enhances the informational benefits of research. 

3]vidin 9 a ani w 2 orJ 0nS K^* S0lve + .«. rtain b a sic problems: problems of 
dividing work bringing activities together to achieve goals 
motivating members to participate through rewards and sanctions' 

S l a C S-r 1VeneSS ' and ^ol^g conflict. Resolving these 
problems stabilizes organizations and clarifies decision-making 
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authority within them. It enforces accountability and provides checks 
on the accummulation of power at the top. Many of the so-called 
"evils" of bureaucratic development, such as formal hierarchies, 
formal rules and regulations, and the specialization of tasks are 
intended to insure equity and efficiency. Nevertheless serious 
constraints may be imposed on policy research by bureaucratic 
characteristics such as the tendency to replace formal agency missions 
with organizational goals, to engage in power struggles with 
,, competing ,, organizations rather than to cooperate, to encourage 
rigidity and territoriality as the arbiter of the division of labor 
and decision-making process, and to resist innovation and change. 

The last characteristic is most central to the debate. Evaluation 
implies change. Improvement is its purpose. Yet agency admini- 
strators and program operators understandably judge evaluation and 
evaluation methodologies in terms of bureaucratic risks and 
benefits— i.e. what is most conducive to the internal maintenance of 
the organization and to its security outside. These practitioners 
genuinely appreciate the need to acquire information on a regular 
basis to judge the value of the programs for which they are 
responsible. For this purpose they need timely, useful information 
geared to their administrative and managerial needs and obligations. 
But information that is likely to suggest that their programs are far- 
less effective than they should be, or possibly ineffective, is 
information that entails serious risks for an organization's leader- 
ship. At the worst, these risks are starkly economic: budgets will 
be cut. Therefore the decision to evaluate is nearly always a 
courageous one, and the choice of evaluation methods within that 
decision must be weighed against many other considerations. 

Given that the decision to evaluate has been made, randomization is 
again the main issue in the debate over the use of quasi-experimental 
vs. experimental designs. Managers properly view service assignment 
decisions as their responsibility. Their service delivery staffs 
derive motivation and reward from the communication of this function 
downward. 

Randomization represents major organization al intervention. This is 
particularly true in JTP, \ f where the careful sorting of clients into 
treatments appropriate tu their individual needs and resources is a 
legislative mandate. 

Historically, randomization procedures have often been subverted f<;r 
this reason, resulting in unknown and unmeasured biases. Random 
assignment must be presented such thut it is viewed by staff as Ljotti 
justifiable and profitable. The price paid for gaining this kind of 
acceptance is a reorientation of staff attitudes and behavior But 
this can reduce the realism and genera? izability of the experiment. 
It would be difficult to apply the final n&s to programs in which the 
intended staff identification with Elective assignment practices has 
remained intact. 
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In summary, organizational and political tradeoffs are a compel ling 
influence in the debate. The intertwining of these considerations 
with scientific concerns amplifies the difficulties involved in 
choosing any one "ideal 11 research design for a particular piece cf 
policy research. 



CONCLUSIO NS 



To a greater extent than we might wish, the beauty of these tradeoffs 
is in the eyes of the beholder. In terms of scientific exchanges, 
there are numerous criteria for judgment: reliability, validity, 
general i zabi 1 ity and replicability— and rigor, real ism and repre- 
sentativeness. And certainly more. With respect to organizational 
and political tradeoffs, feasibility and utility are the significant 
criteria, and a^ , '^.cre ar e many more. There is no one absolute or 
universal crit'' " ror determining the best research design among 
these. One mus : • vomise, choosing the best research strategy to 
fit resources and „iu;ations. 

Understanding some of the advantages and disadvantages of different 
approach^ should lead to better choices, and each kind of approach 
can be improved by efforts to compensate for its specific weaknesses. 
The complexity of choices suggests contributions both evaluators and 
practitioners in JTPA can make to the debate. Evaluators can become 
more knowledgeaole about the supports, problems, and unanticipated 
possibilities in the research environment: 

"The social scientist who desires to undertake policy 
studies, to be an agent of social change and contribute 
constructively to the solution and amelioration of the 
myriad of problems in today's communities must recognize 
that more than technical knowledge and craftsmanship is 
involved. The way he conducts himself, the types of inter- 
personal relationships he develops, and the attention he 
pays to various administrative and organizational tasks are 
equally crucial to the successful fulfillment of his 
role." 7/ 

Practitioners can become more knowledgeable about the scientific 
tradeoffs in terms of the value of the information they provide: 

"Troubles with experimental controls misled even the great 
Pavlov into believing temporarily that he had proof of the 
inheritance of an acquired ability to learn: in an informal 
statement. . .Pavlov explained that in checking up on his 



7/ See Freeman and Sherwood. Social Research and Social Pol i 
page 142. 
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experiments he found that the apparent improvement in the 
ability to learn on the part of successive generations of 
mice v/as really due to an improvement in the ability to 
teach on the part of the experimenter . 11 8/ 

And finally, working together, evaluators and program practitioners 
can consider the circumstances under which quasi-experimental designs 
are more appropriate than experimental designs, and vice versa. 
Anticipating the conclusions of such a joint effort, it is probably 
safe to assume that in evaluating ongoing programs such as JTPA at the 
state and local level, the scientific, organizational, and political 
tradeoffs point to the use of a rigorous quasi-experimental design for 
determining the net Impact of the program. The likely consensus with 
respect to the use of experimental designs is that, whenever possible, 
th y should be used in evaluating one-time or limited pilot and 
demonstration projects at the state and local level, which involve a 
small proportion of the client population of JTPA, and whose specific 
purpose is to test the effectiveness, feasibility, and utility of 
various innovations in program implementation to achieve desired 
changes in client treatments and outcomes: new combinations of 
services or r v service targeting strategies; different program 
emphases with respect to particular subpopulations of clients; new 
state or local prog" am policies or guidelines; or other ideas designed 
to improve state and local training policies and programs. In a very 
different context, at the national level, it is expected that the 
tradeoffs will more likely support experimentation. 

An important concern, in any case, is to obtain cross-national , 
cross-state, and cross-SDA comparability with respect to the variables 
and relationships selected for study, the way these are measured, the 
manner in which the characteristics of different methodologies are 
def i ned , and the extent to whi ch the i nf ormati on obtai ned can be 
generalized beyond a sinjle SDA, state, or cluster of experimental 
national sites. 

If such joint efforts, and such information Comparability and utility 
can be adopted a. the overriding goals, the debate can pleasantly 
continue: more as a stress reduction exercise than as a potentially 
divisive influence in policy research. 



8/ See Kish. "Representation, Randomization, and Control 11 
page 269. 
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SUPPLEMENT A 
DEFINITION OF CONCEPTS 



CONCEPTS 



DEFINITIONS 



Characteristics of the goals of 
evaluation : 

RELIABILITY 



Reliability is related to the accuracy of measurement 
and the faithfulness of the data collection effort to 
the research design. If measures of a given variable 
do not reliably represent the variable as it was de- 
fined, and if these measures are not collected in a 
consistent manner, the information available for 
analysis will not be reliable. 



EXAMPLES 



In collecting data on JTPA services proviso to clsnts, the 
evaluator must be assured that nformation p ^.ach desired 
characteristic of the treatments—the mix, s? nee, length, 
etc. -has been gathered in a consistent way c *curately 
r°corded for each research subject. If data ,o i - action 
guidelines are violated, the information will V 
reliability. 



VALIDITY The interpretation of evaijation information is 

valid to the extent that the data which were analyzed 
to provide conclusions accurately represent the 
definition of the variables studied, Also, the 
definition of the variables must, reflect their true 
qualities, 

Validity is also a function of the extent to which 
other sources of potential error and bias have beer, 
reduced. The research framework and design may be 
well chosen and specified, and the data collected 
reliably, but the results may still reflect un- 
controlled biases that reduce the credibility of 
the findings. 



In studying the gross -oart of JTPA, using an appropriate 
research design and a i »» emphasis on a rich arr^y &f 
outcomes, the findings -uggest that women benefit 
from clasrroom training h ■ I'm men, particularly ri 
respect to increased motirtu to pursue a career pa-*'.. 
"Career path" is difficult i.: ; jre in quantitative 
terms. If measurement error !, nation from the true 
qualities of a variable) is s« .Mtf.ial with respect to 
outcomes such as these import. • • . ther* will be ft 
lesser degree of validity. 



GENERAL1 ZABI LIT Y 



REPLICAB I L IT Y 
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An evaluation may be carefully implemented but its 
results may not be applicable beyond a limited segment 
of the intended subjects of the researclr-or [ eyond 
the particular kind of context or environment in which 
the evaluation was carried out. 



This trait involves the ability to repeat a 
particular evaluation (its framework and methodology) 
at a different point in time and in another kind of 
setting, using similar research subjects-- in order to 
determine the durability of the original findings and 
to further check on their validity. 

If replication is not possible, we can never be certain 
that our results will hold up when the evaluation is 
repeated elsewhere. Our inferences just might be an 
artifact of where and under what circumstances we 
carried out a particular evaluation. 



The use of an experimental design which uses JTPA 
volunteers as the study group, and which of necessity 
interferes with the service assignment process intended 
in the program, will yield results which cannot be 
generalized beyond an atypical group of JTPA clients in 
a program implemented significantly differently than 
intended. 



If an evaluation o! the net impact of JTPA on minority vs. 
nonminority enrolled in Chicago has utilized an experimental 
design, and yielded results that suggest that on-the-job 
training produces better outcomes for minorities, it will 

the ^edibility these findings if a replication 
o* Ja analysis in Seattle prconces similar resu's. 
/.cessful subsequent replication in New York City will 
further "validate" these conclusions. 
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CONCEPTS 



DtFINiHOHS 



EXAMPLES 



Characteristics of the evaluation 
approach, or framework : 

EVALUATION ISSUES Evaluation issues are the research questions to be 

studied in an evaluation. Ihey convey expectations 
about the kinds ol relationships that exist between 
certain factors, or variables, which describe these 
issues. These questions can take the form of 
hypotheses to be tested, propositions to be in- 
vestigated, or simply ideas to be explored. 

In hypothesis testing, and in many investigations of 
formal propositions, the valuator focuses on CAUSAL 
relationships between various influences and effects. 
In other evaluations, the emphasis may be more on 
what kinds of ASSOCIATIONS exist between such variables, 
i.e. , how they may be correlated with one another. 



In JTPA we are interested in two major sets of evaluation 
issues: 1) how JTPA services and other formal interventions 
affect, or are correlated with a number of desired out- 
comes, and 2) how the unique features of JTPA program 
implementation affect, or are associated with those 
outcomes. 



MEASUREMENT Measurement is a process which progressively defines 

the factors or variables being interrelated in study- 
ing evaluation issias. This process begins with what 
are referred to ac 'operational" definitions and 
moves toward one or more specific measures, or 
indices which "stand for" these variables. The data 
elements Ascribing variables in MISs are examples 
of such indices. 

The accuracy of measurement refers to how well the 
selected me^'jros represent the influence or effect 
lo be stud ; * it. 



In JTPA, critical measurement tasks involve precisely 
defining the characteristics of; 1) the program "treat- 
ments", or unique interventions the program provides, and 
the "outcomes" expected to take place as a result of these 
interventions; 2) the characteristics of a program's 
"implementation" which may— in addition to its inter- 
ventions-influence outcomes; 3) the characteristics of 
the program "environment" which can effect outcomes, and 
4) the characteristics of potential "unintended treatments" 
which can explain outcomes. 



Charac teristics of the 
Evaluation nelhotiology : 

tf?RO?«ATENESS' 



This refers to how feasible and efficient the 
research design is, i.e., the guide the evaluator 
selected for sampling research subjects, and 
collecting and analyzing information about them. 
Feasibility is related to how well the design fits 
the research situation; efficiency has to do with the 
degree to which it controls for unknown or unmeasured 
influents that may explain what is observed just as 
well or better than the primary ones being studied. 



In JTPA the use of an experimental design is inappropriate 
for btudying the influence of program implementation. 
This is due to a variety of political, organizational and 
research design problems. For instance, to study the 
effect of a single characteristic of implementation, one 
would have to randomly assign that characteristic across 
a large number of SDAs or project sites. It is obviously 
not possible to do this. 
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CONCEPTS DEFINITIONS EXAMPLES 



The rigor of a research design is the extent to which 
the design yields accurate explanations of the re- 
lationships hypothesized, proposed, or expected, 



The ^ideal" 'experimental design is considered to be the 
most rigorous research design. However, rigorous non- 
experimental designs also involve a high degree of control 
in research situations where experimental designs are 
considered inappropriate. 



CONTROL Control involves procedures which reduce the effect 

of variables that "compete with" program treatments 
in explaining expected relationships, thereby per- 
mitting the evaluator to concentrate on the direct 
link between the program's intended treatments and 
the observed effects. 



Some of the factors which need to be controlled in 
evaluating JTPA implementation or impact are the 
demographic, economic, and labor market characteristics 
of the program's environment which change over tin*. 



Controlling for these extraneous factors, or those 
whose influence the evakator would like to eliminate, 
is a necessary element of an adequate research design. 
The purpose of control: is to reduce error and bias 
which can invalidate evaluation findings. 
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SUPPLEMENT B 
SOME SOURCES AND KINDS OF BIAS 



SOURCE 



KIND 



ELABORATION 



Those associated with being 
part of an evaluation . 



Client's attitudes and behavior, related to their 
participation in a special project which affords them 
a different kind of assistance and attention from that 
received on the average by the typical program client. 



Staffs' attitudes and behavior, related to being part 
of an evaluation which involves compliance with 
policies and procedures that ■*/ not bi the same as 
for other staff, or which involves being studied 
themselves. 



Retesting of clients or staff, which involves their 
being studied more than once over a period of time 



ihe way the evaluation is conducted, in terms of the 
evaluator's interaction with staff, clients, employers, 
and relevant others. 



The evaluator's own personality and behavior. 



The influence of interventions introduced by an 
evaluator's funders. 



Clients are immediately aware of being "studied", if only 
from rumors that they are part of an evaluation, or through 
more frequent or more detailed follow-up surveys than other 
clients experience. This "test effect 4 ' can produce an 
an impact of its own, which is difficult to separate from 
the effects of the services clients receive. This is known 
as the "Hawthorne Effect". 

Staff tend to behave differently when they know they are 
being observed or directly studied. They receive greater 
attention, which may be rewarding, but they are often 
more closely monitored, which can be frustrating. This 
influences the very process that is being studied- 
particularly in evaluations of program implementation. 
These influences are difficult to measure and hard to 
isolate. 

If measurements are made at regular paints in time over 
an extended period, and the client or staffperson is 
aware that data are being collected, the effects of the 
first data gathering can affect the information collected 
n subsequent points. The individual becomes familiar 
with the approach, the data collection interview or 
questionnaire, and the context of the evalutaion. This 
affects responses and may bias the information being 
collected. 

Anything that effects the environment of the evaluation 
can be confused with the services in explaining client 
or employer outcomes. Again it is difficult to quantify 
these nuances, and separate them out from the more direct 
relationship between services and their effects. 

This may support or constrain "normal" behavior on the 
part of those being studied, posing a possible bias. If 
the evaluator's attitudes and general style are consistent, 
at least the bias is a systematic one, and may be easier 
to define. 

Some funders are intrusive, requiring frequent on-site 
monitoring of an evaluation effort as well as continuing 
consultation. This can introduce biases that Utach to 
the attidues and behavior of the grant officer. 
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S0U| M KIND ELABORATION 



Those a ss ociated with the 
"explanatory" framework 
within which tfre research 
issues are to be studied. 



Inadequate selection and definition of the key issues, 
due to the use of an underdeveloped or erroneous 
knowledge base. 



Use of an explanatory framework which relies in- 
sufficiently on the conclusions of previous competent 
research in that issue area. 



Selection of variables and relationships which are 
not the most critical ones, given the questions 
to be answered in the evaluation. 



If the evaluator selects the wrong, or the less relevant 
variables and relationships to study, the evaluation will 
provide less than useful information. For example, in 
process evaluations organizational theory will be more 
appropriate than economic theory in selecting cogent 
issues to study. 

If an evaluator carrying out a net impact evaluation of 
JTPA fails to study the relevant precious research on 
employment and trining programs over the past two decades, 
the evaluator may tail to identify the main issues to 
study. Particularly in the case of experimental or 
rigorous nonexperimental studies, which usually involve 
a small set of variables, it is critical that the 
previous literature be utilized. 

In process evaluations, certain organizational variables 
are more essential than others, given the distinctive 
features of JTPA. 



Those associated with the 
measurement of key issues , 
influences, and effects. 
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Selection of definitions and measures which do 
not reliably represent, these factors, 



The use of too few measures for a variable, or 
too little "averaging out 11 of errors and biases 
in developing indices for variables. 



Development of inadequate measuring procedures. 



Insufficient compliance with adequate measuring 
procedures. 



An absence of repeated measurements on the same study 
sample. 



If the treatments, or JTPA services, are not defined 
precisely, the relationships between treatments and out- 
cor s cannot be accurately evaluated. 

The data elements which are selected to represent the 
variable may be insufficient in terns of affording 
multiple measurement of the variable* Multiple measures 
help to detect and compensate for this type of error, 
and to describe different aspects of a complex in- 
fluence or outcome. 

So*)* times the criteria the evaluator uses in developing 
definitions and indices fail to produce accurate measures. 

The evaluator may decide to use a number of indicators of 
employment as an outcome variable. The criteria for 
selecting them may be based on an adequate knowledge 
base and appropriate procedures, but as the evaluation 
progresses the use of the indicators may be inconsistent 
from one service contractor to the next* 

Although collecting data on subjects has its own effect, 
one-time-only measurement can produce biased information. 
In most impact evaluations, an ideal set of data collection 
points are: prior to receiving services; during program 
participation; at termination from the program; and at 
some point beyond termination. 



SOURCE 



ELABORATION 



Lack of reliability among the different measures 
used to study a variable. 



Inadequate identification and measurement of 
variables other than the program's interventions. 



Those associated with the 
definition of the target 
group to which evaluation 
findings are to be generalized . 



Those associated with the 
development of a research 
design which guides the 
collection and analysis of 
information. 



Failure to use a representative sample of the target 
grouf! *s the study population. 



Failure to adequately conUjJ for the major "competing" 
explanations of outcomes, sixb as: 

1. SELF SELECTION BIAS: due rr differences in the 
characteristics of eligible, 1 ; «!io apply for the 
program vs. those who do not. 



2. STAFF SELECTION BIAS: due to a nonrandora selection 
of applicants into the program. 



3. FAILURE TO ADHERE TO RANDOMIZATION PROCEDURES 
IN EXPERIMENTAL STUDIES. 



If the evaluator wants to study "work motivation" by 
collecting information on; 1) satisfaction with the 
workplace; 2) personal attitudes about the meaning 
of work quality; and 3) desire to "get ahead", these 
indices need to be highly correlated with one another. 
If not, they may be biased indicators. 

Measurement tasks are not limited to the primary variables 
of interesM.e,, to services and desired outcomes. The; 
also involve the measurement of influences which must be 
controlled in the evaluation, such as labor market 
conditions in the community in which the program operates. 



An experiment may use only volunteer applicants or 
enrol lees, in order to defuse objections to randomization. 
However, volunteers are usually not representative of 
the larger target group of programs, and their different 
characteristics will bias results. Also, to encourage 
sufficient volunteers an evaluator may have to provide 
additional information and incentives, which further 
sets these volunteers apart from the regular target 
group and changes the nature of the program treatment, 



Only a subset of eligible individuals apply for JTPA, 
These individuals may have characteristics that distinguish 
them from other eligibles, such as level of need, degree of 
motivation, acceptance of government assistance, etc. 
These potential differences can bias the results of an 
evaluation whose findings are intended to be generalizable 
to the full target population. 

Staff attitudes and behavior have a great deal to do with 
which applicants are actually enrolled. This is an early 
source of bias in the program. "Creaming" is an example. 

Although randomization of the study sample into treated 
and nontreated groups is an important aspect of experiments 
experimental principles are not always applied stringently. 
If staff are inattentive to randomization procedures, or 
delioerately resist them, the two groups will not be 
statistically equivalent on the average, and comparisons 
between them will be biased, 
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OURCE 



KIND 



ELABORATION 



ERLC 



4. INABILITY TO USE A COMPARISON GROUP IN NON- 
EXPERIMENTAL STUDIES. 



INABILITY TO IDENTIFY AND MEASURE A COMPARISON 
GROUP THAT IS SUFFICIENTLY SIMILAR TO THE GROUP 
GIVEN PROGRAM SERVICES. 



Nonexperlmental studies are used in situations where 
randomization is not possible or appropriate. However, 
extensive control over competing explanations of outcomes 
can be achieved through the construction of a comparison 
group which is as "equivalent" as possible to the treated 
group. The evaluator can u;e one set of clients as a 
control for another set in evaluating the differential 
impact of various services, and can also use statistical 
controls. However, the use of an adequate comparison 
group raises the level of control and reduces bias. 

It is sometimes difficult to locate a group of similar 
individuals who have not received a comparable treatment. 
It Is impossible to be certain that the two groups are, 
on the average, identical in the statistical sense. And 
it may be difficult to obtain the kinds of information 
needed, even when the comparison group ot first choice is 
located. 



6. FAILURE TO TAKE INTO ACCOUNT THE COMPLEX INTER- 
ACTIVE EFFECTS THAT USUALLY OCCUR BETWEEN THE 
PROGRAM'S INTERVENTIONS AND OTHER INFLUENCES 
AFFECTING OUTCOMES. 

7. BIASES DUE TO THE CHARACTERISTICS OF A PROGRAM'S 
IMPLEMENTATION OR OPERATION. 



BIASES DUE TO THE CHARACTERISTICS OF A PROGRAM'S 
ENVIRONMENT. 



Sometimes the evaluator focuses only on the average 
effects of the program's services, without measuring the 
extent to which interactions between services and other 
explanatory factors can jointly affect outcomes. 

The evaluator must be able to tease out the effects of 
service delivery practices from the effects of JTPA 
services. A poorly managed project can obscure potentially 
positive program impacts. 

The availability of certain kinds of training slots and 
jobs may greatly affect client outcomes. Labor market 
conditions vary across time and projects. 



sociated with data 

3n. 



Failure to follow systematic data collection principles 
and procedures. 

Failure to systematically track nonrespondents for 
follow-up purposes, and retain information on them 
that explains why they are non-respondents and what 
their past program statuses are. 



Data collection efforts need to be standardized to assure 
reliability. 

Basing results on nonrepresentative follow-up samples 
introduces serious biases. 



sociated with data 



Inappropriate analysis methods, given the purpose 
of the research, the issues to be studied, the kind 
of design used, and t;>e assumptions made by the 
evaluator. 



Statistical methods vary with the purpose of the 
evaluation and can range from simple comparisons of 
means and proportions to complicated multivariate 
analyses. 
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SOURCE 



KIND 



ELABORATION 



Those associated with the 
interpretation of evaluation 
results. 



United issues and variables for study, 



Drawing conclusions from a comparison of inferences 
from one evaluation with those from nonconparable 
evaluations. 



In both experimental and nonexperimental evaluations, the 
kinds of issues and influences studied constrain what can 
be learned. If not enough of the significant factors 
are studied, the results will have Halted utility. 

If an evaluator treats nonconparable evaluations AS IF 
they were replications of his/her research, the con- 
clusions will not be valid or useful. For example, 
comparing evaluations of JTPA with evaluations of CETA 
are risky, since in some cases the differences in the 
findings are explainable on the basis of 1) differences 
in the study samples, 2) the Issues addressed, 3) the 
research designs used, 4) the reliability and validity 
of the data, and 5) the characteristics of the research 
sites. 



Using the establishment of statistical significance 
between program interventions and outcomes as the 
major single criterion of a successful study, ignoring 
the STRENGTH of these relationships, 



Nonrepresentativeness of evaluation sites. 



"Fine-tuning" the results of nonexperimental studies 
of a particular program by referring to experimental 
findings on that program, 



In an evaluation of the effect of job search assistance, 
the investigator way find that it has a different effect 
for men than for women which is statistically significant. 
But this significant difference may be so slight as to be 
irrelevant for policy purposes, 

In both experimental and nonexperimental evaluations, the 
general i lability of the results may be reduced by the 
limitation of the studies to only selected sites, which 
are neither randomly selected nor selected with the 
specific purpose of representing the range of sites 
within a state or across the country. For example, 
the selection of sites in experimental studies may 
favor those in which administrators and program operators 
are willing to randomize clients, or to those which 
have successfully submitted unsolicited proposals to 
funders to study a set of issues of special interest to 
those particular funders. The use of such selection 
methods results in evaluations which are difficult to 
generalize beyond potentially atypical environments. 

Such "calibration" is inapproprate, since experiments 
and nonexperiments involve quite different research 
conditions. Their sampling of program target groups 
differs, as do their data analysis methods. Program 
implementation and program environments are affected by 
them differently. Consequently they produce noncomparable 
information. However, general insights from informal 
assessments of the differences or similarities in their 
findings may be useful in identifying evaluation issues 
that require better or further study. 
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SUPPLEMENT C 
A RESEARCH DESIGN CONTINUUM 



TYPE 



KIND 



PURPOSE 



EXAMPLE 



Nonexperimental 
Designs 



Exploratory Designs 



TO EXPLORE: the win purpose is to formulate 
new ideas and develop and refine research 
questions for more rigorous study. Therefore, 
such designs demand that the evaluator become 
thoroughly immersed in the object of study 
and rely heavily on insight and intuition, 
since dependence on specific hypotheses or 
a relatively small set of significant 
variables is neither possible nor feasible. 
In fact, this is where one develops and 
refines possible hypotheses for further 
study. These designs are nest appropriate 
when little information is available from 
previous research! or the evaluator wants 
an in-depth look at a complex phenomenon. 

Evaluations using exploratory designs range 
from unstructured to highly structured 
studies* The major advantage is 
flexibility. 



Participant-observation or case studies are very 
useful in studying program implementation, such 
as staff attitudes and behavior. Such exploratory 
designs can contribute important insights about 
the experiences of clients in JTPA programs. 



Descriptive Designs 
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10 DESCRIBE: The purpose is to investigate 
associations among the major explanatory 
variables short of attributing cause-effect. 
These designs are well suited to issues for 
which there is already a body of information, 
but rigorous quasi-experimental or experi- 
mental approaches are not appropriate. These 
designs permit a range of data collection and 
analysis procedures, but are not as flexible 
as exploratory designs. They require careful 
planning to economize the research effort, 
protect against bias, and increase the 
data's reliability and validity. This 
demands a clear definition of what 
variables are to be measured and why, 
and what objects will be sampled and 
how they will be selected. 

Again descriptive studies can vary from 
simple but well -organized descriptions 
of phenomena to rigorous systematic 
surveys. However, they do not control for 
bias as well as quasi-experimental 
designs. 



Social surveys, such as structured systematic 
post-program follow-up surveys of JTPA clients 
and employers, are examples of this kind of 
design. Descriptive studies of JTPA gross 
outcomes also illustrate this kind of approach. 
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PURPOSE 



EXAMPLE 



Quasi-experimental 
Designs 



TD ESTABLISH CAUSE-EFFECT: the purpose is to 
closely approximate the principles and methods 
of experimental designs, in the absence of 
randomizing clients Into treated and un- 
treated groups. These designs range from 
more to less rigorous alternatives. The 
most rigorous are effective in reducing 
sources of significant bias, such as the 
effects of major influence! other than 
the treatments. However, this spectrum 
of designs involves various compromises 
between feasibility and "ideal" experi- 
mental design. In general, these designs 
require a valid theoretical model of why 
and how a program works so that cause and 
effect can be inferred. 



Studies of the nature of associations or 
correlations between JTPA services and outcomes 
without the use of a comparison group, il lustra 
such designs. More rigorous studies of causal 
relationships, using a constructed comparison 
group, are also examples of this approach. 



Experimental Designs 



TO ESTABLISH CAUSE-EFFECT; the purpose is 
to make optimal use of scientific method in 
providing the most accurate and valid infor- 
mation for testing cause-effect explanations 
of relationships between interventions and 
effects. These designs accomplish this 
through a combination of randomization, the 
use of a control group, and statistical 
controls for bias. At their best, experi- 
mental designs yield the most valid 
inferences. But experimental designs also 
range from more to less rigorous, as the 
evaluator seeks the "ideal 11 . 



Field experiments, such as are being recowende- 
for studying JTPA at the national level, are 
examples. Evaluations of previous national pil 
and demonstration projects have often utilized 
such designs. Host evaluations of ongoifij 
programs, however, have not used experimental 
approaches. 
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